To start off with LimeSurvey, let’s try and create our first small survey, that should make you familiar with the most common functions in LimeSurvey. We’re going to create a survey in which we do the all-famous wug test after we have asked some details on our participants. Follow the steps below to create it.

1. Log in to the administrator environment.
2. Create a new survey. Call it “Wug test” or something similar. Set the base language to English (note that this can not be changed later). Add some description, a welcome message and an end message on the General tab. The other tabs you can disregard for now. Don’t forget to click Save.
3. Create two question groups: one for personal information of the participant, and the other for the actual wug test.
4. Create questions to ask for a participant’s age, gender and handedness. Make sure you add them to the correct group. Try to use question types that help your participant to fill in correct data. E.g., for asking for a participant’s age, you probably want to use the numerical input. Hover over a question type to see an example of the question type. Test your survey and try different inputs to see whether everything works.
5. Further refine your personal questions by clicking on the Show advanced settings button. For example, for the participant’s age, you will want to have your participants only to fill In integers, and ranging from a certain age, to a certain age. You might want to disable the No answer option in the gender question by making it necessary.
6. Use the Relevance equation field to only display the handedness question when the participant is younger than 21. You can use the expression age <= 20 for that, given that age is the question code for your age question. When you test your survey, you’ll find that the question will disappear when you enter an age higher than 20. If you want the question to hide up front, you have to add && age != 0 to the expression, as the default for a numerical input is 0.
7. Create a question in the wug test question group. Find a nice image of a wug online and add it to your question using the image uploader, top right in your editor. Be sure to use the Upload tag and to click Send it to the server. You might also use the direct URL to the image you’ve found (but then you should make sure the image stays where it is, of course). You can set width and height and even add borders or alignments.
8. Use the advanced editor (top left in your editor) to create some nice text layout, e.g., change the text font and size or even add some HTML or JavaScript if you’re familiar with that.
9. Test, test, test. Don’t forget to test your survey on different browsers (Internet Explorer, Mozilla Firefox, Google Chrome) and screen resolutions (consider tablets as well!). And while you’re at it, try and break your survey by adding nonsense answers (e.g. 19.999 as your age, this is called monkey testing). If you can actually break your survey, try to find advanced settings that don’t break your survey anymore (e.g. only allow integers in a specific range, or use a drop down list with age ranges).
10. You’re done! You’re now the happy owner of your own wug test!

Randomization of questions (within a question group)

LimeSurvey allows for randomization of questions with so-called “randomization groups”. This allows you to add all questions that should be in a randomized order into the same group (you can use whatever names you like here). The option can be found in the advanced settings for every question. Per question group you can create as many randomization groups as you would like.

Randomization of question groups

Randomization of question groups is not possible in the current version (1.92) of LimeSurvey installed on our servers. Starting from version 2.0 you’re able to randomize the order of question groups. For now, a possible solution is to work with several randomization groups within a question group (which unfortunately will lead to having all questions on one page). Another idea would be to use separate surveys and add a landing page which directs participants randomly (ask lab support for a how-to). A more complicated, though clean approach is available via relevance equations.

Showing visual stimuli

Adding images to your questions is actually quite easy. In the editor, the rightmost button allows you to select either an image already available on the server or a public URL, or to upload images yourself. The Image Info tab allows you to set the width and the height of the image, as well as borders and alignment.

Showing auditory stimuli

Adding audio to your questions is somewhat more complex. Note that the display of an audio component is often browser- or even plug-in-dependent. Using the new HTML5 audio standard, instead of examples you may find online, is therefore a better idea. For example, create a “list (radio)” question and add the following as an answer option:

<audio controls autoplay>
  <source src="https://upload.wikimedia.org/wikipedia/en/4/41/Ascending_fifths.wav" type="audio/wav">
  </source>
</audio>

This should create an audio fragment that starts automatically when you load the question (caveat: you should be using a modern browser). The “controls” and “autoplay” options can be deleted if you wish. Read up on the audio element above for more information (you can e.g. find there how to loop the fragment). The sound file itself can be uploaded via the image uploader (see the paragraph on visual stimuli above) or shared via e.g. Dropbox. You can also read our own how-to for more information on audio in LimeSurvey.

Scales

Likert scales and semantic differentials

Likert scales and semantic differentials are a common way to visualize answers in scaling questions. In LimeSurvey, the end points of the scale can be created quite easily: you can use the pipe symbol “|” to discern the left and right ends of your scale in array question types. See also this page from the manual.

Thurstone scale

For a question using the Thurstone scale, you can use a the Ranking question type. This question type allows your participant to rank the answers.

Output of results to SPSS, Excel or R

LimeSurvey allows for outputting your results to SPSS, Microsoft Excel or R for further analysis. In a started questionnaire, go to the Responses and Statistics tab. There, you’re able to view your results, some basic statistics on them, and you can also export your results to your favorite statistics program. The export to R might sometimes be somewhat buggy, a workaround is then to use .csv-files as output.

Conditional display based on answers to preceding questions

You can use Relevance equations to ensure that some questions are not asked, given answers to previous questions. E.g., if someone answers to a question that he is a student, you might want to ask for his current major, but if not, your participant can skip this question. You can use relevance equations for this purpose.

The expressions you can use in the relevance equation are really like programming. You can use the common relational (==, >, !=), logical operators (&&, ||, !) and even some functions here, and referral to questions and answers is done by their respective codes. Here, your expression would probably something like “answer.student”. See this page in the manual for more details on the possibilities and sample surveys.

If you’re not really feeling comfortable using expressions, you can also use conditions, which can be modified per question by clicking the corresponding icon. This will give you a nice (though limited) graphical interface for adding relevance equations.

Assessment

LimeSurvey allows for assessment. You can assign scores to answers, and then use these scores to calculate an end total. To enable assessments, edit the general settings of your survey, tab Notification & Data management. For each question type that allows assessment, the answer possibilities know have a field in which you can set the score. You can show the score and/or a message after the survey, using the assessments overview in your survey menu. Refer to the LimeSurvey manual for tips and tricks.

A known issue is that for arrays, only one score is possible per answer possibility. While this is sometimes exactly what you want (e.g. answers according to a scale), for some types of questions (e.g. true/false) you will want to have a separate score per question. An issue has been created in the LimeSurvey bug tracker for that. To fix this now, you will have to use a workaround: create separate questions and deal with the assignment there.

Permissions

You can enable your fellow researchers to modify your survey. To do so, hover over the button to edit your survey (second menu from the top, third bottom from the left) and take the third option: Survey permissions. There you can add users and/or groups to your survey. By default, users you add don’t have any rights, so that’s what you can alter in the next step. You’re able to give pretty fine-grained rights to a user; you can e.g. limit a user to only viewing the survey, and not adding or modifying any questions.

To make your life easier, we provide a downloadable list of (basic) questions. These question templates are created for surveys in Dutch. To import them in an English survey, see the last section of this chapter. LimeSurvey also has its own repository of questions.

Provinces

A simple list of Dutch provinces, with their abbreviations as codes. The flags of the provinces appear in the answer options. You can download it here (right-click to save). There’s also a drop down option without the flags available here.

Age ranges

Ask a participant’s age. The participant can select from age ranges (5 year intervals). Created from the English version downloadable from the LimeSurvey repository. You can download it here.

Importing Dutch questions into English surveys

If you download a Dutch question template, but your survey is actually in English, importing a question will not seem to work. That’s because of the language settings. A workaround for that is as follows:

1. Open the question in a text editor.
2. Use find/replace to change all occurrences of <![CDATA[nl]]> to <![CDATA[en]]>.
3. Also change <language>nl</language> in the top of the file to <language>en</language>.
4. Import the question. The question should now be imported correctly.
5. You can now change the question and answer labels to your liking.

Of course, this method basically works to and fro any language.

There are basically three ways for your survey to get participants: open it for all internet users, open it only to a limited number of preselected participants, or make the survey only available for administrators. The second option requires tokens to work.

Open access

Setting a survey to be open for all users is the default way of managing surveys. While you don’t have control over the participants (basically anyone can enter results), this setting does allow for the most responses, as you can e.g. use social networks to spread your survey. On the other hand, you might have to prepare yourself for nonsense answers (or find some way to filter these out of your results).

Token access

If you have a closed group of participants that you would like to test, tokens are the ideal solution. Tokens allow you to send a specific link to your participants, that only they can use to enter their answers. This allows you visibility on who completed the survey, and also prevents people from submitting to your survey without your permission. See the manual for a guide how to set up tokens.

Administrator access

When there are no access rules defined, but your survey has started, you are still able to enter results as an administrator of your survey by starting the survey from the main survey menu. You can do this by clicking the green gear, see the image below. Do make sure you’re survey is activated, otherwise the results won’t be saved.

This mode is useful when you want participants to only use your device, for example when you have an external document or video that you want your participants to read or see before or after taking your survey, or you’re running your survey directly before or after an experiment (and don’t want participants to start the survey afterwards).

Participant quotas

Often you have a certain number of participants in mind for your survey, and if that number is reached, you’re not interested in more data. Or, based on of your questions, you will want a certain number of participants for option A, and a certain number of participants for option B. Both of these situations can be handled by participant quotas.

The interface to add quotas can be found in the main editing menu of your survey, second menu from the top, third button from the right, hover over that, and find Quotas. You can add a quota, and after that, you can specify whether a certain answer to a question should be taken into account for that. Do note that not all question types are supported for dealing with quotes (e.g., free texts are not, for obvious reasons). Further information (and examples) on quotas can be found in the LimeSurvey manual.

When you are creating your survey, you might run into problems with formatting. You would love to have your questions just one pixel to the right, or the answers two pixels to the left. The question editor usually does not always suffice then (although you might get creative with non-breaking spaces), and you will want to edit the main template. That is surely possible, but you will need some extra access privileges. Ask for them at ICT&Media!

In the template editor (the painting symbol in the top menu), you can edit your template as you please. The best procedure to work is to copy one of the default templates and work from there. Give your template a descriptive name and make sure you set it as template for your survey. The fun then starts. The template contains of three types of files: .pstpl (internal LimeSurvey format), .css (Cascading Style Sheets) and .js (JavaScript).

For most lay-out tweaks you will want to edit the .css-files, and specifically the template.css file. To find which id or class you will have to change, you could use either Moazilla Firefox’s or Google Chrome’s “Inspect Element” when you right-click on the element of your liking. Add, change or delete the properties in the middle box and check your changes by testing your survey or using the selection box on the right to select the corresponding page.

For examples of template edits, you can check the LimeSurvey manual. And of course, lab support is happy to help you, even though this is a bit outside their comfort zone.

There are some things that LimeSurvey does (yet) miss, and which you might find necessary for your survey. We’ll discuss these limitations in this chapter.

Easy way to add a lot of (similar) questions

There’s currently no option to easily add a number of similar questions, e.g. using a paradigm in which you have a number of audio files that are each accompanied by the same questions. There are however options to copy questions, question groups or even whole surveys. For questions, this is the most straightforward, as you can use the copy-option when you edit a question. Question groups can be exported and then imported again to create copies. For surveys as a whole, you can also use the export and import functionality, but when you create a new survey, there is also an opportunity to copy an already created survey, by clicking the “Copy” tab.

Measuring reaction time

Measuring reaction time in on line paradigms is far from straightforward. In LimeSurvey, there is a default way to save the responses in seconds (you can select to save timings in the general settings of your survey). There’s even a patch available that allows you to measure the milliseconds (not installed on our server as of now). However, be aware that these measurements are far from accurate: reaction times are highly dependent upon operating system, browser and hardware (keyboard, mouse, monitor) connections. If you plan to do statistics on your reaction times, you had better use an experimental framework like Zep.

Stimulus display and hide paradigm

Although for some questions there are options to hide a question after a certain amount of time, the way this works is not really suitable for a real experiment. Of course, there are possibilities to code your own paradigm, but the out-of-the-box support is limited. LimeSurvey does offer some interesting options however. For example, you can prevent your participants from going back in the survey (this is one of the general settings), so that he is not able to go back and see what the current or previous stimulus would look like. Also, the options for randomization are of course typical for experiments.

LimeSurvey is, as stated above, a pretty extensive framework and thus has quite a learning curve. There are some viable alternatives for LimeSurvey, but these (of course) have their pros and cons as well.

One of the alternatives is Google Forms, which is certainly easy to use, with drag- and dropping to create forms. The most notable disadvantages of using this are limited flexibility (e.g. no randomization) and saving of participant data on a non-university server.

A non-free alternative is SurveyMonkey. The functionality is comparable to (but a bit more professional than) Google’s, but that comes with a price. There is however a free layer available, that might be interesting for creating a small survey, even though options are limited (e.g. you can’t add any question logic in the free version).

Currently, the Faculty of Social Sciences (FSS) is conducting a pilot with Qualtrics. This seems to be a major player in the field of online surveys. We at the ILS Labs are not involved in this pilot; if you are affiliated to FSS you might try to contact the local ICT team. Since Qualitrics is survey software, not experimentation software, it will not be particularly good at things like making multiple lists, etc., but we at the lab have no personal experience with it.

Another option we don’t know much about is Wextor. Wextor does seem to be designed for experiments.

Finally, here’s a link to Theo Veenker’s WWStim. This link is mainly provided for historical reasons; cgi is not allowed to run anymore on the Humanities server, so you won’t be able to use it here. If you were to install it on a server elsewhere, this would require some knowledge of web servers. Advantages of WWstim used to be that the script can easily handle a lot of similar stimuli. However, the lack of a clear graphical user interface might well frighten laymen.

“Database error: Too many columns” when activating a survey

This is an unfortunate error that signals your survey contains a bit too much questions. The error is described in this forum post. Especially multiple-choice questions seem to take up a lot of space in the database, so you might consider choosing another question type. Another work-around is to split your survey into two (or three) surveys (each survey will be saved in a separate table). This bug seems to have been fixed in a newer version of LimeSurvey (2.0), but we’re not yet ready to update to that version.