Experiment Guide
To create a new alfred experiment we use the script.py file, which contains the experiment code in every alfred experiment. You can download the script.py with a simple experiment template from here. To run this experiment locally, you will need a working alfred installation (click here for the alfred installation guide) and two additional files which can be obtained here for the config.conf and here for the run.py. In this tutorial it is assumed that you already have a working installation of alfred. If you do you can simply download the three files above, move them all into an arbitrary directory (but make shure they are all in the same directory!) and start the example experiment right away by executing the run.py file. Local alfred experiments are always executed through the run.py file.
TODO: Upload alfred installation guide
Here is a screenshot of what you should see when you launch our example experiment:
Our example experiment consists of three pages containing only simple text elements. These are the examples we will use to get a first impression of how alfred experiments are scripted. Now let's have a look at the scripty.py file which contains the whole experiment script. You will learn about the other files later. Just bear in mind that the run.py must be executed in order for our experiments to launch. To open the script.py I recommend you use a Python IDE which means a development environment that specializes in displaying and working with python code. I recommend PyCharm, which is available for Windows, MacOS and Linux and which comes with all basic features like syntax highlighting, code completion and code inspection. You can download the free community edition of PyCharm here.
Once you have opened the script.py with the editor of your choice you will see this code:
# -*- coding:utf-8 -*-
u'''
Example experiment script using Alfred - A library for rapid experiment development.
Experiment name: Exemplary name
Experiment version: 0.1
Experiment author: ctreffe <example mail>
Description: You should give a short description of your experiment and it's purpose
'''
#################################
### Section 1: Module imports ###
#################################
from alfred.question import *
from alfred.questionGroup import *
from alfred.element import *
from alfred.helpmates import *
import alfred.settings as settings
from alfred import Experiment
#################################################
### Section 2: Global variables and functions ###
#################################################
expName = 'Basic Experiment'
expVersion = '0.0'
#################################
### Section 3: Custom classes ###
#################################
class MyCustomQuestion(CompositeQuestion):
def __init__(self, **kwargs):
super(MyCustomQuestion, self).__init__(**kwargs)
self.addElement(TextElement(u'Basic text example added on initialization', fontSize='huge'))
def onShowingWidget(self):
if not self._hasBeenShown:
self.addElement(TextElement(u'Basic text example added on showing of question', fontSize='huge'))
def onHidingWidget(self):
self.addElement(DataElement(variable=u'This is added to data on hiding of question', name=u'exampleString'))
########################################
### Section 4: Experiment generation ###
########################################
class Script(object):
def generate_experiment(self):
exp = Experiment('qt-wk', expName, expVersion)
# First we generate a simple question containing only one text element:
myFirstQuestion = CompositeQuestion()
exampleText = TextElement(text=u'Test string for TextElement', alignment='center')
myFirstQuestion.addElement(exampleText)
# We can also add elements to our question by using the 'elements' argument as seen below:
mySecondQuestion = CompositeQuestion(
elements=[TextElement(u'Test string for another text element')]
)
# We can also use a custom made question which generates its' own elements:
myCustomQuestion = MyCustomQuestion()
# Questions can be pooled in question groups, which can add certain functionality to experiments:
myQuestionGroup = SegmentedQG()
myQuestionGroup.appendItems(mySecondQuestion, myCustomQuestion)
# Before an experiment can be launched we must always add our content to it:
exp.questionController.appendItems(myFirstQuestion, myQuestionGroup)
return exp
generate_experiment = Script().generate_experimentAs you can see the script starts with a multiline docstring which contains important information about the experiment contained in this script.py. You should always take your time to give some information about your experiment in this docstring as it will make it a lot easier for you to keep track of your experiment scripts.
The following code is divided into four sections. I will give you a brief introduction into each section:
In this section we use python's import command to enable the alfred framework within python and to get access to the different alfred components. You might wish to import other python modules later but for now the basic alfred modules are sufficient to create simple experiments.
You can define veriables and functions which you can access anytime and from anywhere in your experiment. This section provides the space for these "global" variables and functions. As you can see I have added two simple variables already. expName and expVersion are strings which i will use in section 4.
Attention: Global variables should be handled as immutable meaning that you should only access but not change them after initialization! In general you should refrain from using dynamic global variables. If you need data to be exchanged i.e. between different questions in alfred you should access them through the alfred DataManager class, wich will be introduced here.
TODO: Introduce DataManager and access to question data
In this section you can create customized content for alfred, for example elements, questions or question groups with advanced functionality or special layout properties. Custom classes are also used to access data at runtime which is an essential requirement if you want to add dynamic content to your experiments. I have added a custom question class to our fresh script.py file but for now we will start to expand our example experiment with static content. Therefore you can ignore this section at the moment.
The fourth section of our script.py file is the most important part of our experiment script since it contains the function which will actually generate the experiment and tell alfred what to do with our content. This function is named generate_experiment and contained within a class called Script.
Attention: You are not allowed to change the structure of this sections, i.e. by renaming the script class or the generate_experiment function. You only need to work and insert your code within generate_experiment !
Before we can start to add content you need to learn about the general structure of alfred experiments. We define three
types of classes, which are displayed in the following diagram:
Elements are the most basic components of alfred experiments. Elements contain the content which experimenters want their participants to see and to interact with. Elements can consist of nothing more than simple text like in the TextElement class which I used in the example script or they can be complex input elements like the LikertMatrix class. We will learn more about different types of elements throughout this tutorial.
For a complete reference to all alfred elements and their arguments click here.
Questions are the main components of alfred experiments. One question equals one page in the experiment. We can display multiple elements in a single question, i.e. some instruction text followed by a group of likert items we want our subjects to answer. Experiments in alfred are always displayed question by question which means you have to use the navigation bar to switch between questions (e.g. by clicking on 'forward'). The diagram above shows an experiment containing 7 different questions which are displayed sequentially to our participants. The numbers in the upper left corner of each box show the questions' order of appearence. Alfred contains different question classes which provide you with special functionality but 99% of the time you will use the standard question class CompositeQuestion.
Todo: Question reference
As you can see in the diagram some questions are not assigned directly to the experiment but to different questiongroups. Questiongroups are used to control the way participants navigate through experiments by restricting access to their subordinate questions. For example, the SegmentedQG allows participants only to move forward through it's subordinate questions one at a time. Participants are not allows to navigate backwards ans therefore cannot return to questions once they completed them. The standard questiongroup class which is simply called QuestionGroup has no restrictions at all regarding the navigation of it's subordinate questions. Participants are allowed to move back and forth and even to jump between questions in both directions which allows them to leave questions out and to return to any question from anywhere. Questiongroups can be nested in other questiongroups which allows us to create complex navigation schemes for any experiment.
Attention: The main experiment class in alfred is a normal QuestionGroup class itself and participants can move freely between any content that is directly added to an experiments QuestionController class (see below for further explanation).
Todo: Questiongroup reference
Suppose we want to expand our experiment with a new element to ask our participants for their gender. This new element should appear on the very first question of our example experiment. To achieve this we will have to take a closer look at the generate_experiment function in section four of our script.py file. The basic layout of section four looks like this:
class Script(object):
def generate_experiment(self):
exp = Experiment('qt-wk', expName, expVersion)
# Insert experiment code here!
return exp
generate_experiment = Script().generate_experimentThe only things you are allowed to change in these lines of code are the arguments in parentheses inside the initialization call of the basic Experiment class. There are three arguments, which must be provided in order for an alfred experiment to launch:
- experiment type: This must be a string (no unicode!) containing 'qt', 'qt-wk' or 'web'
- experiment name: This must be a string containing the experiment's name
- experiment version: This must be a string containing the experiment's version
As of now it is not necessary to introduce you to the different types of experiments. Almost all of our local experiments will be using the qt-wk type which is very similar to the web experiment type that we use to create online experiments. In fact you could just change the string from 'qt-wk' to 'web' and insert the contents of our script.py file into a new experiment in our webbased control interface for online experiments (mortimer) without any further editing. The old qt experiment type is only needed on very few occasions and you will not need to learn about this right now.
Experiment name and version are needed to keep track of your participants datasets which are stored in an online database where you can access your experiment's complete data file (also through mortimer). I use the global variables I created earlier to assign a name and a version number to our example experiment. This way it is easier to keep track of my names and versions (because i don't have to search for the line in generate_experiment but can easily lock at the top of my script.py file).
For the next part I recommend you use an editor which shows line numbers since I will refer to parts of the experiment code in terms of lines in the script.py file. If you look inside the generate_experiment function the first command is a call to the main Experiment class in line 58. This call initializes the whole alfred framework and provides us with the core alfred modules:
- QuestionController: Controlling the content and navigation in experiments
- DataManager: Providing access to experiment data
- UIController: Controlling the display and layouting of questions and elements
- SavingAgentController: Controlling data storage to databases and local files
- MessageManager: Providing communcation interface to participants
- ExperimenterMessageManager: Providing communcation interface to experimenters
You will learn about the different modules as you progress through this tutorial. Nevertheless most of these modules need not to be accessed by experimenters directly but work automatically (e.g. the SavingAgentController and the UIController). For our experiment the most important module is the QuestionController because this class provides every experiment's root questiongroup to which experimenters must add the content they want to be displayed.
In lines 60-66 we create our first question containing only a simple text element. Therefore we first create the question by calling the CompositeQuestion class in line 62. Questions have a lot of arguments experimenters can use but as you see you can simply call the question without any arguments (empty parentheses) so that the question will take it's standard configuration. In a second command we create a simple text element with an example string to be displayed within our question. I used the alignment argument to center the text horizontally. The last command in line 66 assigns the new text element to our question. By using the addElement method contained in any question class we inform the element about the question it belongs to and we also inform the question about it's new subordinate element. Our new question now contains a simple text element.
myFirstQuestion = CompositeQuestion()
exampleText = TextElement(text=u'Test string for TextElement', alignment='center')
myFirstQuestion.addElement(exampleText)Attention: Not every question allows for elements to be added. Only questions derived from CoreCompositeQuestion class contain the methods and properties needed to display elements! At the moment there are only very few custom made questions which are not derived from CoreCompositeQuestion but keep in mind that you might get to know questions that work without elements.
As said earlier we want to add another element to our first question asking participants to state their gender. I will use a simple TextEntryElement for this task. We can just insert the following line of code into our script.py at line 65:
gender = TextEntryElement(instruction='please state your gender', name='gender')After this we need to assign our new element to the first question by changing line 66 like this:
myFirstQuestion.addElements(exampleText, gender)First of all notice that I have changed the called method from addElement to addElements since we need another method if we want to add multiple elements at once. With addElements it is possible to assign more than one element to a question and we add gender just after exampleText. The elements' order if appearence is determined by the order of any elements being added to a question. You could add the gender element before the exampleText element to display it first on your question page. If you start the example experiment now you should see your newly added TextEntryElement:
Classes can be initialized using arguments which are confugiration variables used by the initialized class. Like Experiment, which is called with an experiment type, a name and a version, all of alfred's main classes can be called with various arguments only some of which are mandatory. The CompisiteQuestion class for example can be initialized without any argument while the TextElement class must have at least a text to be displayed. You can check our reference section to see which arguments are available for any alfred class.
Todo: Finish reference section
mySecondQuestion = CompositeQuestion(
elements=[TextElement(u'Test string for another text element')]
)We can also use an argument to assign elements to suitable questions in alfred. The CoreCompositeQuestion class provides the elements argument which can be a list of elements to be displayed in a question. In lines 68-72 of our example script.py file I created a second Question with a simple text element by using the elements argument. Python allows us to put the arguments of an class into multiple lines and also the items in our elements list could be written in multiple lines. Brackets and parentheses are the important sign telling python where a list of arguments or items does end. As you can see i put the closing parenthesis in line 72 for better clarity and structure of my code. Using the elements argument you don't have to assign your elements later with an extra command. This can save you some effort but also tends to be the less comprehensible code structure. We can easily add some more elements to the elements list:
mySecondQuestion = CompositeQuestion(
elements=[TextElement(u'Test string for another text element'),
TextEntryElement(instruction='Test instruction', name='testItem'),
HorizontalLine()
]
)As in any python lost we can write our elements in multiple lines but still we must adhere to the necessary syntax, dividing elements with commas and opening and closing our list with brackets. See the following minimal example of different ways to write a list in python for better understanding of what I did above:
list = ['element','element','element']
# the first list equals the following list. The only differences are linebreaks and additional whitespace!
list = ['element',
'element',
'element'
]Be aware that the order of display for elements is always determined by the order they are added to a question. As you can see there is more than one method to assign elements to a question and later on we will learn about the internal order of their execution so we will know when which elements are added by the different methods. For now you only need to now that the display order of elements is determined by the order you add them to a question.
myQuestionGroup = SegmentedQG()
myQuestionGroup.appendItems(mySecondQuestion, myCustomQuestion)Like the CompositeQuestion class the three central types of questiongroups in alfred (QuestionGroup,SegmentedQG, HeadOpenQG) can be initialized without any arguments as you can see in line 80 of our example script file. After creating a new questiongroup we can add questions or other questiongroups by using the appendItem method for single items or the appendItems method for multiple items. In our example script we are adding tweo questions to myQuestionGroup and so we use the appendItems method in line 82. Following I will give you a brief description of the three basic questiongroup classes:
- QuestionGroup: Allows participants to move and jump freely between subordinate questions without checking for mandatory answers. Every experiment's root question group is a unrestricted questiongroup so that you will need to add your content to a subordinate question group if you want to restrict participant navigation.
- HeadOpenQG: Restricts participants to only move forward question by question with answer checks but allows them to move and jump backwards as much as they want. This way participants can return to any question that was already completed and jump or move back to the last uncompleted question, which we call headquestion. But advancing past the headquestion is only possible stepwise question by question.
- SegmentedQG: This questiongroup restricts participants to move forward question by question through it's subordinate questions. Participants are not allowed to jump in any direction or to move backwards. This is the normal navigation mode for many of our experiments.
In the example experiment I added two questions to a SegmentedQG questiongroup. This means that you can only move forward through this questiongroup and you will not be able to get back to the first question after completing it. You will, however, be able to return to the last question in this questiongroup as it won't ne closed down after completing the last question. We intend to change this behavior in future verisons of the alfred framework. For now you will need to remember that users will have the ability to return to the last question of a SegmentedQG if it is nested in a standard QuestionGroup (also the root questiongroup in QuestionController) or a HeadopenQG. Therefore I often use multiple nested SegmentedQGs to restrict users from moving backwards to any last question of a preceding questiongroup.
Notice that you can create complex navigation schemes (for example being able to return to every instruction page but nowhere else) by nesting multiple questiongroups. You are also not restricted in the levels of nesting.
After I created three questions (you will learn about the third customized question in the next step of this tutorial) and structured them with a questiongroup in our example experiment it is finally necessary to add all this content to the experiment itself. As stated earlier in this tutorial we must access our experiment's root questiongroup which is contained in the QuestionController class. We see this in line 86 of our example experiment:
exp.questionController.appendItems(myFirstQuestion, myQuestionGroup)We access the QuestionController through our experiment, which is the object i called exp in line 58, and then use the standard questiongroup methods appendItem or appendItems to add questions or other questiongroups to our root questiongroup.
Attention: Alfred experiments won't start if the root questiongroup is left empty or if you add no questions to it or subordinate questiongroups. You must at least have one question in any experiment. Otherwise your experiment will not be started!
Until now you have learned about basic concepts of alfred experiments and how to compose an experiment with questiongroups, questions and elements in the generate_experiment function. All content which is created this way will be added before participants start to work through the experiment. Since this content cannot be changed after being added to the experiment we call it static content. Aside from the possibility to add filters for questiongroups, questions and elements you will not be able to have your static content react to user behavior or specific user input.
Filters are custom functions you can use to determine if an element, question or questiongroup should be shown or hidden for a specific participant. From within filter functions you can access experiment data and choose if some content should be displayed or not. Filter functions have to follow a strict syntax:
def filter(experiment):
#insert funtion code here
boolVar = True # or False
return boolVar
myQuestion.setShouldBeShownFilterFunction(filter)
myQuestion.removeShouldBeShownFilterFunction()First of all we define a new function called "filter" (see first line). Filter functions will be called from within alfred with only one argument. This argument will be the current experiment object therefore the function takes "experiment" as argument. You can name the argument differently but keep in mind that you can use this variable to access your experiment and all of it's modules. Inside your filter function you can add code to determine if a specific content should be displayed to your participant. You don't have to use the predefined experiment argument if you don't need to. We will learn more about the possible contents of filter functions in another section of this guide. For now you only need to know that filter functions must return a boolean variable meaning they must return True or False. If you want a speccific content to be displayed your function must return True and if you want the content to stay hidden your function must return False. The function I created always returns True as you can see above.
Attention: Since filter functions are called from within alfred with the current experiment as only argument your function must have exactly one argument, regardless if you want to access your experiment data or not! Filters with no arguments or more than one argument will cause a program exception and your experiment will crash.
After creating a suitable filter function you can add this function to elements, questions or questiongroups by using their setShouldBeShownFilterFunction as you can see in the 8th line of code above. This method takes your custom filter function as argument and adds it to it's class. From then on everytime a participants navigates to this content the function will be called to determine if it should be displayed or not. For example, if a filter function determines that a specific question should not be shown this question will be skipped. If an element should not be shown this element will stay hidden if it's parent question is being displayed.
Attention: Filter functions can be used multiple times in different content items. You do not need to create a new filter function for each item. Just assign the function to multiple content items.
If at some point of your experiment a filter function becomes oblivious you can always remove it from a content item using this item's removeShouldBeShownFilterFunction method as seen in the example above.
Filter functions provide the ability to control which content a participant will see and which content will be hidden according to this participants response behavior but since filter functions must be defined before any user interaction takes place they don't equate to real dynamic interaction with a participant. Filter questions are also used in paper and pencil questionnaires and provide only a very restricted amount of dynamics.
In many experiments you might want to display dynamic content that directly reflects previous participant input or behavior. In simple cases you might just want to display an earlier input to a participant but in more complex examples you might want to create diagrams depending on participant performance or even simulate a computer partner to work with your participants and react properly to their actions. In all of these cases you need to generate content directly depending on you participants' input which cannot be achieved in the generate_experiment method since it allows only content to be added before any user interaction takes place.
Therefore alfred provides experimenters with the ability to dynamically generate content at runtime but before we can use this feature we need to understand how and when different methods of alfred experiments are executed. To demonstrate the sequential execution of an experiment I created the following diagram:
As you can see the first method executed, that adds content to an experiment, is generate_experiment. When a new content item is created in generate_experiment python will call this item's initialization method which is called init. Executing init will create and configure the associated class and accordingly content item. Init is a standard method for any python class and we will take a closer look at it later in this guide. Notice that adding content in the generate_experiment method will cause the init methods of all content items to be executed which happens before any user interaction. After all static content items are created, alfred calls the experiment's start method and participants can begin to work on the first question. As participants navigate from question to question using the navigation bar alfred will always call different methods of the currently active question. Before any question is displayed alfred executes this question's onShowingWidget method. When a participant uses the navigation bar to exit any question alfred will call this question's onHidingWidget method. Any participant interaction with a question always takes place between the associated onShowingWidget and onHidingWidget methods, chronologically speaking. After completing the last question in an experiment alfred executes the finish method which will close an experiment.
To generate and display content dynamically at runtime we can create custom questions and insert code into their onShowingWidget and onHidingWidget methods. In section 3 of our example experiment I created a custom question class called MyCustomQuestion:
class MyCustomQuestion(CompositeQuestion):
def __init__(self, **kwargs):
super(MyCustomQuestion, self).__init__(**kwargs)
self.addElement(TextElement(u'Basic text example added on initialization', fontSize='huge'))
def onShowingWidget(self):
if not self._hasBeenShown:
self.addElement(TextElement(u'Basic text example added on showing of question', fontSize='huge'))
def onHidingWidget(self):
self.addElement(DataElement(variable=u'This is added to data on hiding of question', name=u'exampleString'))As you see in line 1 of the above sample to create a custom question we create a new question class derived from one of alfred's standard classes. In my example I create a new custom question class which is derived from CompositeQuestion, the standard question class for alfred experiments. After creating our new class we can define new class methods to overwrite the original methods included in the parent CompositeQuestion class. Our new custom question will be identical to it's parent except for those newly defined methods. This way for example it is very easy to create a new question which creates a new TextElement just before it is displayed. You only need to create a question class derived from CompositeQuestion and then overwrite the original onShowingWidget method with your custom code. Every other property and method of the custom class will still be identical to the parent class it was derived from meaning that you don't need to rewrite the whole class but only the parts that need customization.
There are two specific aspects in the example above which you need to be aware of. First, it is also possible to overwrite a question's init method for example to generate content on initialization right inside a question. This can be helpful if you want to use specific questions in different experiments. Instead of copying the code from your generate_experiment method you can just copy & paste a custom question from one experiment to another. This can help you to structure your code more clearly. Overwriting the init method can serve mutliple purposes and a clear and comprehensible code structure is only one of them. Notice that the actual name of the init method is _init_! Since alfred executes essential commands in a question's init method you need to be very careful if you want to overwrite it. First of all you must call the init method with the additional **kwargs parameter. This will catch all arguments that are used to initialize your question in generate_experiment and make them available for further usage. Secondly, you need to execute the essential commands from a parent question class's init method manually by calling it's original init method as you can see in line 3 of our example. The super command allows you to execute a parent class method which you have overwritten. By calling the init method of our parent CompositeQuestion we can make sure that the original alfred commands needed on question initialization will be executed. Be aware that sometimes you might want to execute the super command at a specific point in your custom init method, e.g. when your custom code needs to be executed before the normal initialization procedure is started. When calling super we must use the **kwargs parameter again to transfer all the arguments used on question initialization, which are normally processed by CompositeQuestions' init method, to the parent's init method. This way all arguments we use on initialization of our question in generate_experiment are evaluated and processed as they are supposed to.
The second aspect you need to consider when creating custom questions is that the onShowingWidget and the onHidingWidget methods are both called every time a question is being displayed or being hidden. This means that all your custom code, e.g. adding some dynamic text elements, will be executed multiple times if a subject chooses to return to your custom question. In most cases this behavior is wanted with the onHidingWidget method since we often use this method to compute data and create additional variables which can simply be overwritten if onHidingWidget is executed multiple times. Nonetheless, some scenarios for the onHidingWidget method and in most cases for the onShowingWidget method you want your code to be executed only the first time a question is shown or hidden. Otherwise some content might be created multiple times without intention. If you want your custom code to be executed only once when a question is shown or hidden for the first time you will need to check if the question was already shown or hidden. You can achieve this by using the _hasBeenShown and _hasBeenHidden properties of any question class in alfred derived from it's standard Question class. I have already used the _hasBeenShown property in the custom question example above. Here is a small example of using _hasBeenHidden in the onHidingWidget method:
def onHiding(self):
if not self._hasBeenHidden:
self.addElement(DataElement(variable=u'This is added to data on hiding of question', name=u'exampleString'))You can see that the usage of _hasBeenHidden in onHidingWidget is identical to _hasBeenShown in onShowingWidget. By using a simple if statement you can assure that your custom code will be executed only once. You will have to decide which parts of your code should be executed every time a question is shown or hidden and which parts should be executed only the first time.
Custom questions allow you to compute new variables and react to user interaction at runtime and thus to create truly dynamic computer experiments which is a huge advantage compared to the restricted possibilities you get when using filter functions. To make full use of custom questions you will often need to access your participant data, which is explained in the next chapter of this guide.
As with questions you can also create custom elements and question groups. You just need to create classes derived of one of alfred's standard classes and then overwrite their methods with your own custom code. However, the customization of elements and question groups requires advanced knowledge of programming in python and of the alfred framework's internal structure. From experience I can say that custom elements or question groups are required only in few scenarios and therefore they don't lie within the scope of this guide.
As I stated in the last chapter we often want our experiments to react directly and dynamically to user input at runtime. Custom questions allow us to execute commands at runtime but to react to our participants' behaviour we also need access to their input data. Accessing and evaluating participant data at runtime is one of the core requirements of dynamic experiments and alfred provides us with methods for accessing and managing the current participant's dataset through the DataManager class which is a submodule of every alfred experiment.