1. Qualitative Research
1.1. Positivism - It states that all knowledge must be based on logical inference from a set of basic observable facts. Positivists are reductionist, in that they study things by breaking them into simpler components. This corresponds to their belief that scientific knowledge is built up incrementally from verifiable observations, and inferences based on them.
1.1.1. Positivists dominate natural sciences, although nowadays they are called post-positivists. Due to the fact that they tend to accept the idea (due to Popper) that it is more productive to refute theories than to prove them, and we increase our confidence in a theory each time we fail to refute it, without necessarily ever proving it to be true.
1.1.2. Precise theories - Positivists prefer methods that start with precise theories from which verifiable hypotheses can be extracted, and tested in isolation.
1.1.3. Reductionism - Note that a belief in reductionism is needed to accept laboratory experiments as valid in software engineering — you have to convince yourself that the phenomenon you are interested in can be studied in isolation from its context.
1.1.4. Research methods - Positivism is most closely associated with the controlled experiment; however, survey research and case studies are also frequently conducted with a positivist stance.
1.2. Constructivism - Constructivism, also known as interpretivism (Klein and Myers, 1999), rejects the idea that scientific knowledge can be separated from its human context. In particular, the meanings of terms used in scientific theories are socially constructed, sointerpretations of what a theory means are just as important in judging its truth as the empirical observations on which it is based.
1.2.1. Theories verification - Constructivists concentrate less on verifying theories, and more on understanding how different people make sense of the world, and how they assign meaning to actions. Theories may emerge from this process, but they are always tied to the context being studied.
1.2.2. Social science - This stance is often adopted in the social sciences, where positivist/reductionist approaches have little to say about the richness of social interactions. Constructivists prefer methods that collect rich qualitative data about human activities, from which local theories might emerge.
1.2.3. Research methods - Constructivism is most closely associated with ethnographies, although constructivists often use exploratory case studies and survey research too.
1.3. Critical Theory - It judges scientific knowledge by its ability to free people from restrictive systems of thought (Calhoun, 1995). Critical theorists argue that research is a political act, because knowledge empowers different groups within society, or entrenches existing power structures.
1.3.1. Reseach approach - Critical theorists therefore choose what research to undertake based on whom it helps. They prefer partici- patory approaches in which the groups they are trying to help are engaged in the research, including helping to set its goals. Critical theorists therefore tend to take emancipatory or advocacy roles.
1.3.2. Focus on minority groups - In sociology, critical theory is most closely associated with Marxist and feminist studies, along with research that seeks to improve the status of various minority groups. In software engineering, it includes research that actively seeks to challenge existing perceptions about software practice, most notably the open source movement, and, arguably, the process improvement community and the agile community.
1.3.3. Research methods - Critical theorists often use case studies to draw attention to things that need changing. However it is action research that most closely reflects the philosophy of critical theorists.
1.4. Pragmatism - It acknowledges that all knowledge is approximate and incomplete, and its value depends on the methods by which it was obtained (Menand, 1997). For pragmatists, knowledge is judged by how useful it is for solving practical problems. Put simply, truth is whatever works at the time.
1.4.1. Embrace relativism - This stance therefore entails a degree of relativism: what is useful for one person to believe might not be useful for another; therefore truth is relative to the observer. To overcome the obvious criticisms, many pragmatists emphasize the importance of consensus — truth is uncovered in the process of rational discourse, and is judged by the par ticipants as whatever has the better arguments.
1.4.2. Research approach - Pragmatism is less dogmatic than the other three stances described above, as pragmatists tend to think the researcher should be free to use whatever research methods shed light on the research problem. In essence, pragmatism adopts an engineering approach to research — it values practical knowledge over abstract knowledge, and uses whatever methods are appropriate to obtain it.
1.4.3. Research methods - Pragmatists use any available methods, and strongly prefer mixed methods research, where several methods are used to shed light on the issue under study.
1.5. Grounded Theory - It is a technique for developing theory iteratively from qualitative data. I such process, the data is analyzed without and preconceived categories. By the time patterns emerge, the data is compared with the existing information, and more data is collected to refute or support the emerging theory.
1.5.1. Relations with philosopichal stances - Despite its close association with the con- structivist stance, Grounded Theory probably approximates how most scientists end up developing theories. The difference is that Grounded Theory makes the process explicit and systematic.
2. Research Questions
2.1. Exploratory questions - should be made in the early stages of a research, in order to help understand the phenomena, and indentify useful distinctions.
2.1.1. Existence questions - Questions focused on identify the existence of something. E.g "Does X exist?
2.1.2. Description and Classification questions - Questions focused on describing, categorizing, and measuring something. E.g "What is X like?", "What are its components?", "How it can be categorized?"
2.1.3. Descriptive-Comparative question - Questions focused on identifying the difference and similarity between one thing to another thing. E.g "How does X differ from Y?"
2.2. Base-rate questions - Since you have a clearer understanding of the phenomena and know the ideas about its nature, you should ask base-rate questions to understading the base situations of you phenomena, to identify what is normal or unsual.
2.2.1. Frequency and distribution questions - Questions focused on discovering quantitative aspects of the phenomena, such as it frequency, and amount. E.g: "How often does X occur?", "What is an avaregae amount of X?
2.2.2. Descriptive-Process questions - Questions focused on understanding how the phenomenon occurs, which is its steps.E.g "How does X normally works?, "What is the process by which X happens?", "What are the steps X goes through as it evolves?"
2.3. Relationship questions - It should focus in the relationship between two different phenomena, and wheter the occurence of one is related to the ocurrence of the other.
2.3.1. Relationship questions - The questions can be, e.g "Are X and Y related?, "Does ocurrences of X correlate with the occurrences of Y?"
2.4. Causality questions - After relationship questions, it is natural to try to explain and identify how the releantionship of two things can have a cause and effect scenario.
2.4.1. Causality questions - Questions related to find whether one thing causes another thing, or what causes that thing. In SE we usually ask if a particular tool, method or technique improves quality, speed, and so on. E.g "What causes Y? ", "Does X cause Y?"
2.4.2. Causality-Comparative questions - Question that investigate relationships between different causes. E.g "Does X cause more Y than does Z?"
2.4.3. Causality-Comparative Interaction questions - Questions that investigate how context affects a cause-effect relatonship. E.g "Does X or Z cause more Y under one codition but not other?"
2.5. Non-empirical research questions - Also called design questions, it focuses on better ways to do software engineering.
2.5.1. Design questions - The questions can be, e.g "What's an effective way to achieve X?". "What strategies help to achieve X?"
3. Research Methods
3.1. Controlled Experiments - Its a research method that investigates a testable hypothesis , where one or more independent variables are manipulated on controlled enviorinmentas t measure their effect on one or more dependent variables. Such method allow the investigation of cause-effect releationship between N variables.
3.1.1. Preconditions of a Controlled Experiment - A precondition for conduncting an experiment shoulde be a clear hypothesis. The hypothesis should guide all steps of the experimental design.
3.1.2. Define the population and a representative sample - In controlled experiments the hypothesys needs to be prove for the whole population by testing it on a representative sample.
3.1.3. Control the process - Variables other than the chosen indepdent variables must not be allowed to affect the experiment.
3.1.4. The Experimental method points to positivism - Since experiments reduce the complexity, by allowing only few variables of interest to vary in a controlled environment.
3.1.5. Experiments are theory-driven - It causes a strength because the researcher can base the hypothesis on derived theories, although it causes weakeness since it decide which variables to ignore, which disregards real life conditions
3.2. Case studies - Its a empirical method that investigates a phenomenon whithin its real-ife context. Case studies offer in depth understaing of how and why certain phenomena occur, and can reveal the mechanisms by which cause-effect relationships occur.
3.2.1. Types of case studies - Exploratory & confirmatory case studies
3.2.1.1. Exploratory case studies are used as initial investigations of some phenomena to derive new hypotheses and build theories.
3.2.1.2. Confirmatory case study are used to test existing theories.
3.2.2. Preconditions of a case study - It should have a clear research question concernd with how or why certain phenome occur.
3.2.3. Selection of cases - Case study research uses purposive sampling rather than random samplling. The researches should focus to select cases that are most relvant to the study proposition. Based on it, the case study method can consist of a single case study or multiple case studies.
3.2.3.1. Single, or unique case study - Its used when only a single case study is needed, due to the fact it can be a critical case that holds the whole theory, or is expected to yield interesting insights about what happens in crisis/critical conditions
3.2.3.2. Multiple case study offer greater validity, since it suits better for replications. Such way can be used for literal replication that focus to show the same results in multiple cases. Also, it can be use for theoretical replications that expect to show contrasting results.
3.2.4. Define a unit of analysis - The unit of analysis can be a company, a project, a team, an individual developer, a specific event or product. It is important to define to ensure the study focuses on the intended phenomena. The unit should have data extracted from interviews, obersavions, and so on.
3.2.5. Case studies are more suitable in situations where the context is expected to play a role in the phenomena
3.2.6. Weakenes of cases studies - The major weakness of case studies is that the data collection and analysis is more open to interpreation and researcher bias.
3.2.7. Cases studies fit with all fou philosopical stances - Different stances can affect the way in which cases are selected and the data analysis is performed.
3.3. Survey Research - É usada para intedificar características de uma grande população de indivíduos. It can be conducted through strucutred interviews, data logging techniques, and so on. The bases characteristic of a survey is the selection of a representative sample from a well-defined population, and the data analysis techniques to generalize the findings.
3.3.1. Preconditions for a survey - is a clear research question that asks about the nature of a particular target populations. Its important to define the unit of analysis to determine the appropriate sampling technique
3.3.2. Types of survey designs - Cross-section design & Case-control design
3.3.2.1. Cross-sectional design focus on obtaining a snapshot of participants' current activities.
3.3.2.2. Case-control design asks each participant about several related issues in order to establish whether a correlation exists between them, across the population
3.3.3. Major challenge in surveys - The major challenge is the sampling bias. Sampling bias can cause problemes in generalizing the survey resulsts, due to the fact that the respondents to the survey may not be representative of the target population.
3.3.4. Challenge related to the question design - Ensure that questions are designed in a way that yields useful and valida data is hard. Its difficult to phrase the questions such that all participants understand them in the same way.
3.3.5. The Survey method points to positivism - due to the desire to characterize an entire population via sampling techniques. It relies in a belief of reductionism, and a concern with generalizable theories.
3.4. Ethnographies - Etnography is a research method that focus on the sociology of meaning through field observation. The goal is to study a community ofpeople to understand how the members of that community make sense.
3.4.1. Ethnography study benefits - Such method does not impose preconditionsit avoids imposing any pre-existing theories, but instead focuses on how the members of the community themselves make sense of their social and cultural setting. The researcher explicitly considers his/her own pre-conceptions and how they influence understanding of the studied community..
3.4.2. Preconditions of Ethnography - it include a research question that focuses on the cultural practices of a particular community, and access to members of that community.
3.4.3. Paritipant observation - Its a special form of ethnography where the researcher becomes a member of the community being studied for a period of time. Here, the researcher is not trying to understand the community via the observations of an outsider, but rather through the privileged view that comes from membership.
3.4.4. Ethnography points to constructivist stance - Underlying ethnographic research is the idea that members of a community construct their social and cultural practices on the fly, and their perceptions of those structures also define them. Because of that stance, ethnographic researchers don’t seek to prove hypotheses and theories, but rather create local theories to improve understanding.
3.4.5. The major challenge of Ethnography study - The biggest challenge in ethnographic research is to perform detailed observa- tion, data collection and analysis while avoiding preconceptions
3.5. Action research - In Action Research, the researchers attempt to solve a real-world problem while simultaneously studying the experience of solving the problem. action research is a relatively new idea, and there is widespread discussion about appropriate methodology, and even debate on the validity of action research as an empirical method.
3.5.1. Preconditions for Action research - is to have a problem owner willing to col- laborate to both identify a problem, and engage in an effort to solve it. In action research, the problem owners become collaborators in the research. In some cases, the researcher and the problem owner may be the same person.
3.5.2. Action research points to critical theory - In an action research project, it is normally taken as self-evident that the problem needs to be solved, and that the adopted solution is desirable: knowledge gained from the research empowers particular individuals or groups, and facilitate a wider change.
3.5.3. The major challenge for action research - its immaturity as an empirical method. Although frameworks for evaluating action research have been proposed (e.g. Lau, 1999), they tend to be vague or subjective, leading to accusations that action research is ad hoc.
3.6. Mixed-Methods Approaches - This approach can be characterized as mixed methods research — a more complex research strategy that emerged in the recognition that all methods have limitations, and the weaknesses of one method can be compensated for by the strengths of other methods. Mixed method research employs data collection and analysis techniques associated with both quantitative and qualitative data.
3.6.1. Challenge of Mixed-Method Approach - While mixed method research is a powerful approach to inquiry, the researcher is challenged with the need for extensive data collection, the time-intensive nature of analyzing multiple sources of data, as well as the requirement to be familiar with both quantitative and qualitative forms of research.
3.6.1.1. The Sequential explanatory - Such strategy is characterized by the collection and analysis of quantitative data followed by the collection and analysis of qualitative data. The purpose of this strategy is typically to use qualitative results to assist in explaining and interpreting the findings of a quantitative study. It is particularly useful when unexpected results arise from the quantitative phase.
3.6.2. Types of Mixed-Methods Approaches -The Sequential explanatory strategy & The Sequential exploratory & The Concurrent triangulation strategy
3.6.2.1. The Sequential exploratory - Such is characterized by the collection and analysis of qualitative data followed by the collection and analysis of quantitative data. Its purpose is to use quantitative data and results to assist in the interpretation of qualita- tive findings. This strategy is also useful for testing elements of an emerging theory resulting from a qualitative study.
3.6.2.2. The Concurrent triangulation - Such strategy is probably the most familiar and widely used among the mixed-method approaches. This strategy uses different methods concurrently, in an attempt to confirm, cross-validate or corroborate findings. Triangulation is motivated by the fact that often “what people say” could be different than “what people do,” and thus collecting data from multiple sources helps improve validity.
3.6.3. Mixed-,Methods approaches fits in any philosophical stances.
4. Data Collection Techniques
4.1. Techniques - the researcher must decide which data collection techniques are the most suitable for gathering data based on the study’s unit of analysis. Multiple techniques can be used to gather data from different perspectives, as there are advantages and limitations to each technique. Indeed, using multiple techniques allows the researcher to triangulate even within a single method.
4.1.1. Select suitable techniques - The researcher must be careful on the techniques chosen. It is important to note the advantages and disadvantages of the different techniques from the perspectives of the experimenter, the participants, the generalizability and reliability of the results.
4.1.2. Careful blend of techniques should be chosen. A careful blend of techniques can help to offset potential bias and leads to a more comprehensive understanding of the research topic. New researchers should ensure they are familiar with the techniques they select, and that they are aware of the potential pitfalls they may face.
4.1.3. Apply a pilot test - it is always advisable to pilot-test the data collection instrument, and to pilot-test not just the collection aspect of the instrument, but also the analysis procedure. Many problems do not arise until some data is analyzed and it is often possible to detect such prob- lems with even a small data set.
5. Empirical Validity
5.1. Positivists validity - The key steps include deriving study propositions from the theory, designing the study to address the propositions, and then drawing more general conclusions from the results. Each of these steps must be shown to be sound.
5.1.1. Threats to validity - In reporting positivist empirical studies, it is important to include a section on threats to validity, in which potential weaknesses in the study design as well as attempts to mitigate these threats are discussed in terms of these four criteria. This is important because all study designs have flaws. By acknowledging them explicitly, the researchers show that they are aware of the flaws and have taken reasonable steps to minimize their effects.
5.1.1.1. Construct validity - It focuses on whether the theoretical constructs are interpreted and measured correctly. Problems with construct validity occur when the measured variables don’t correspond to the intended meanings of the theoretical terms.
5.1.1.2. Internal validity - It focuses on the study design, and particularly whether the results really do follow from the data. Typical mistakes include the failure to handle confounding variables properly, and misuse of statistical analysis.
5.1.1.3. External validity - It focuses on whether claims for the generality of the results are justified. Often, this depends on the nature of the sampling used in a study.
5.1.1.4. Reliability/Conclusion validity - It focuses on whether the study yields the same results if other researchers replicate it. Problems occur if the researcher introduces bias, perhaps because the tool being evaluated is one that the researcher herself has a stake in.
5.2. Constructivists validity - Many researchers who adopt this stance believe that the whole concept of validity is too positivist, and does not accurately reflect the nature of qualitative research. That is, as the constructivist stance assumes that reality is “multiple and constructed,” then repeatability is simply not possible
5.2.1. Eight strategies for improving validity of constructivist research defined by Creswell (2002)
5.2.1.1. Triangulation - use different sources of data to confirm results and build a coher- ent picture.
5.2.1.2. Member checking - go back to research participants to ensure that the interpreta- tions of the data make sense from their perspective.
5.2.1.3. Rich, thick descriptions - where possible, use detailed descriptions to convey the setting and findings of the research.
5.2.1.4. Clarify bias - be honest with respect to the biases brought by the researchers to the study, and use this self-reflection when reporting findings.
5.2.1.5. Report discrepant information - when reporting findings, report not only those results which confirm the emerging theory, but also those which appear to present different perspectives on the findings.
5.2.1.6. Prolonged contact with participants Make sure that exposure to the subject population is long enough to ensure a reasonable understanding of the issues and phenomenon under study.
5.2.1.7. Peer debriefing - Before reporting findings, locate a peer debriefer who can ask questions about the study and the assumptions present in the reporting of it, so that the final account is as valid as possible.
5.2.1.8. External auditor - The same as peer debriefing, except instead of using a person known to the researcher, find an external auditor to review the research procedure and findings.
5.3. Critical theoristis validity - assessment of research quality must also take into account the utility of the knowledge gained. Researchers adopting the critical stance often seek to bring about a change by redressing a perceived injustice, or challenging existing perspectives. Repeatability is not usually relevant, because the problems tackled are context sensitive. The practical outcome is at least as important as the knowledge gained, and any assessment of validity must balance these.
5.3.1. Lau criteria - It offers one of the few attempts to establish some criteria, specifically for action research. His criteria include that the problem tackled should be authentic, the intended change should be appropriate and adequate, the participants should be authentic, and the research- ers should have an appropriate level of access to the organization, along with a planned exit point. Most importantly, there should be clear knowledge outcomes for the participants.