Experimental Design

This repository aims to provide the auxiliary material used during the experiment reported in our paper [1]. In particular, here we make publicly available the set of questions asked during the semi-structured interview of our exploratory case study.

Questions

The text in bold is a description for the researcher, only the questions below are meant to be asked.

Stakeholder identification:

• When you need to make a decision that affects other stakeholders (either within Airbus or outside), do you always know clearly the person you need to contact to make every decision?

• [If yes] Is that information captured explicitly somewhere? This is, if for some reason (you were to be promoted or win the lottery and leave the company) and someone else would take your position, would this person know who to contact?

Tools:

• Do you use tools when making decisions? (dashboards, excel sheets, proprietary tools, etc.)

Need for support to make the process fairer:

• Is there a hierarchy in which the person at the top makes the decision? Or, on the contrary, decisions are made based on a consensus of different people?

• When more than one person is involved, do you think that people with stronger opinions get things done their way? Is there some voting process in which the opinion of every person has the same weight?

Understanding their decision making process:

• When there is a decision that expands different people and they need to meet, in your experience, how are these meetings organized (i.e., physically)? How many people more or less need to meet? How long does it take?

Efficiency:

• Are there people in those meetings that do not need to be there?

• When you revisit a decision, do the same people need to meet again?

• Is it useful to be more agile when making decisions?

Triggers for revisiting decisions:

• What are common triggers that make you revisit a previously made decision? (Is it data? Feedback from people? New hires? Changes in technology?)

• Are decisions periodically revisited?

• Based on feedback (from people or collected data), do you continuously monitor and revisit your decisions to ensure that they are still valid? If so, to what extent is this automated?

Meeting objectives:

• We assume that you collect and analyse data to help you check whether your objectives are being met (e.g., KPI). To what extent is the calculation of these indicators automated?

Origin of decisions:

• In your opinion, to which extent are decisions based on evidence or personal feelings/intuition?

• [If feelings] Can they be transformed to evidence?

Framework could improve Quality:

Are there cases in which poor decisions were made due to the lack of decision assistance? For instance, having more metrics would have helped make better decisions, contacting the right person would have provided new insights that would have changed the decision for the better…

Framework could improve Efficiency:

Are there cases in which some decisions could have been made earlier if you or your colleagues had had decision assistance?

Framework could improve Traceability:

• Do you need to keep track of people who made certain decisions and the input they used to make their decision?

• [If so] Is this information stored explicitly in documents or any other support?

• Is there a systematic approach to store this information?

Need for the variability model / Potential limitations of the variability model:

• In our framework we assume there is a Variability Model that encodes explicitly the possible options for decisions, i.e., a set of options to choose from, or a specific value that needs to be determined. Can you imagine decisions where this is not the case?

• Is it always possible for you to quantify the impact of the different alternatives/decisions?

• Do you capture the uncertainty associated with your decisions?

• When multiple decisions are being made, how do you ensure that they are in line with the same global strategy?

• What is the decision space and its scope?

• Are you always making decisions in the same context? (involving the same stakeholders, related to the same domain (e.g., manufacturing)), etc. or, on the contrary, do you frequently make decisions out of that context/scope?

• How is defined the scope in which you make decisions?

Application alternatives:

• Do you think that counting with decision-making support would be more useful when used as: 1) A methodological approach, 2) Implemented in a tool, 3) Both? Why?

• Do you think that counting with decision making support would be more useful when used: 1) Within the company, 2) To interact with third-parties, 3) Both? Why?

Feedback (Is our conceptual model missing something?):

• Is there anything else that you take into account for your decision making that we have not discussed?

References

[1] Lola Burgueño, Damien Foures, Benoit Combemale, Jörg Kienzle, Gunter Mussbacher. "Global Decision Making Support for Complex System Development". In Proc. of the 32nd IEEE International Requirements Engineering. 2024.