Version 21.01


Webforms works offline

Previously you could open a webform and then book mark it in order to be able to access that webform offline. Now, in version 21.01, the entire webforms page works offline. If you visit webforms while connected to the network all of the surveys that you have access to will be automatically downloaded and cached. Then if you go offline you will still be able to open these surveys.

Note there are some restrictions with this feature. If you have media files in your survey then these will not be automatically downloaded. If this is the case make sure that you open the survey while online. Once you have done this the media files will have been downloaded and cached and you will be able to access the survey though the webforms page while offline. In addition files required for language translation are not automatically downloaded. Hence you should open at least one survey before going offline to ensure that you have the language file for your language (English, Spanish etc).

A new look for the server

One of the challenges of maintaining a large server application over time is that the software frameworks that are used to manager the user interface change over time and it can be a lot of work to update old pages to match new ones. So the Smap server has web pages implemented using multiple technologies and with different appearances.

This release introduces a concerted effort to make these pages more consistent, simpler and faster. Of course that means introducing yet another technology version and style! In this case Bootstrap 4.5. However all the pages in the administration module along with the console have already been converted. Our aim is to convert and make consistent at least one other page per future release. Sol please bear with us while we work our way through this transition.

We have also standardised on horizontal menus across the top of the page. For no particular reason! Although they do use less screen space than vertical menus.

One advantage of the new pages is that they work much better in mobile phones.

Converted the user management page into two pages

The user management page now only has tabs for aspects of managing users:

  • Users
  • Projects
  • Organisations
  • Enterprises

All the other tabs that were also on that page are now on a Settings page:

Picture of settings page


  • The font colour of the navigation menu can be specified in the appearance tab of the settings page
  • Appearance options for an organisation can no longer be set in the organisations tab. This tab is used by administrators to manage multiple organisations. Instead appearance is set in the appearance tab by administrators of an individual organisation.
  • Setting of the webform banner image has been moved to the webform options tab on the new settings page.
  • Users can be added to a security role in bulk. When creating or editing a role select the “Users” panel to add or remove users.
Bulk editing of users with a security role
  • Add a menu option to access Web Forms from the home page
  • Add an option to select the logo used in a reports for a form. This logo can be specified in the settings dialog in the online editor and is selected from images in shared resources.
  • Add support for embedding Quicksight dashboards into the server

Version 20.10

This month we have revisited a design decision made 8 years ago to restrict each form to having a single geometry (geopoint, geoshape, geotrace). By form I mean the main form within a survey or a subform that can be created using “begin repeat”.

The rationale for this was that shape files used in GIS analysis, and GeoJSON files, only allow a single geometry. All the other values are attributes of that one geometry. So the geometry might specify the location of a building with attributes such as name, number of stories, construction material. In a similar way in Smap each Form and Sub Form is stored in its own database table and this table was allowed at most one geometry with all the other questions being attributes of that geometry. In addition the location was always renamed to “the_geom” which simplified some of the early coding in Smap.

This restriction has now been lifted. You can specify multiple geopoints, geoshapes and geotraces in a single form. The relationship between these geometries and the other questions in the form is up to you and your from design. You may decide that a single geometry per form still makes sense for your project in which case you can create your forms that way.

Multiple geometries in a form

Uploading Surveys from a spreadsheet

No longer will you get an error if you have more than one location in your form. Also location questions will no longer be automatically renamed to “the_geom”. This has the potential to cause problems if you are updating an existing survey that has already collected location data. The issue can happen if:

  • You created a survey in a spreadsheet with a geopoint called say “my_location”
  • You uploaded that survey to Smap version 20.09 or less where its name would have been changed to “the_geom”
  • You then complete some surveys
  • Later you decide to make some change to the survey and you replace the existing survey. In the meantime Smap has been updated to version 20.10.
  • Now your location question will not be renamed and future surveys will record the location under “my_location”.

If the above happens you will see a warning in red text when you replace the survey saying that the location question will have a different name. If that is going to be an issue you can go back to your spreadsheet and change the name of the location question to “the_geom” so that it is compatible with the previously collected data. Then replace the survey again.

Analysis Page

When you create a map view for a survey you can now specify the geometry to be used for each form in the survey. A single geometry from each form will then be displayed on the map.

Dialog showing how the location questions are selected

Exporting Shape, KML and VRT files

When exporting these files you first have to select the form / sub form your are exporting. Now you can also select the location question to use.

Selecting location question for export, note only the questions in the selected form are enabled

Data API

If you specify that the data is returned as geoJson you can now optionally specify the question that will be set as the geometry using “geom_question” in the URL, for example:


Importing Data

Previously when you backed up collect data to a spreadsheet, a geopoint question would be put under two columns “lat” and “lon”. Now each geopoint will still be put in two columns but the name will include the question name so for example “my_location_lat” and “my_location_lon”.

If you have an existing backup with just lat_lon then you can still import that as the system will just grab the first geopoint question that it can find in the form and of course there should only be one in that case.

Start Location

If you set start location in your survey then the location of the start of the survey will be automatically recorded without requiring input from the enumerator.

You can now select this preload to show on a map in the analysis page, export a shapefile based on the location or create follow up tasks at the location.

Note. When creating tasks the location question to be used is found automatically, it is chosen in the following order:

  • First. The first geopoint question answered by an enumerator that is found in the top level form
  • Second. A geopoint value automatically set using the “Start Location” preload
  • Third. The geometric center of the first geoshape or geotrace question that is found in the top level form

Other Changes

  • Users with “View Data” permission can now access reports
  • Fixed a bug where having server side calculations in a survey caused creation of a backup of the survey results in a spreadsheet to fail
  • Add a warning message in webforms if a browser is not supported. Previously some older browsers could just cause webforms to stop

Adding a Quicksight dashboard to Smap

Several customers use Power BI to show a dashboard from Smap data. They extract the data by periodically calling Smap’s data API. This works well, although it can add a significant CPU load to the Smap servers where there is a lot of data and Power BI updates frequently.

I thought I would have a go at using AWS Quicksight to do the same thing. This proved to be straightforward and within a few hours, including the learning curve, I had a working dashboard. which included the donut chart shown below.

Survey submissions by organisation.

Cool things about Quicksight

  • It is serverless so It can scale automatically from small implementations to very large. This scaling also applies to cost which should be small for small usage and large for heavy usage.
  • Dashboards can be embedded into your website
  • You are not restricted to just accessing Smap data you can include text documents, other databases, use artificial intelligence services to gain insights into the data etc.
  • Apps are available for iOS and Android to show the data.
  • It now supports maps which seems to be a relatively recent addition
  • When accessing a Smap server, which is also on AWS, the request does not need to go across the Internet
  • A built in data engine called “SPICE” which stands for Super-fast, Parallel, In-memory Calculation Engine offloads the work from the Smap server for your data crunching.
  • It is pretty easy to use.

How to get Started

  • Create an account on AWS
  • Ask Smap to provide a logon to access your data on the server. Specify to Smap the surveys you need to access as well as any table joins you want. We will set up secure read only access for you to just your data.
  • Start creating dashboards.
  • Embed the dashboards into your own applications


If you are already a PowerBI, Tableau or other BI user then you can use that tool to add a dashboard to visualise Smap data. Probably using the API but we can also consider a solution with a direct JDBC data connections.

If you are not currently using a BI dashboard then you can consider using AWS Quicksight.

Version 20.09

September’s release is now out with a big focus on Webforms and in particular making sure it has the same capabilities as fieldTask.

New Features

lookup and lookup_choices are now supported in webforms

These functions are equivalent to search and pulldata however they only work online as they will make a network request to the server to get data when they need it. “What is this” I hear you say, why would I use them when I can use pulldata() and search() both online and offline. Well if you are going to be using fieldTask in remote areas then you would not use them however if:

  • You expect to always have a network connection and
  • the data you are looking up is very large 100,000 or a million rows or the data is changing very frequently and you want always to be up to date

then the online versions will be much faster.


lookup supports multiple values

As per the “pulldata” function you can now use the “lookup” function to get multiple values from the server while online. This means you can use it to get data that will populate “repeats”.

The following video shows how this can be used to create an educational assessment system.

Video showing use of lookup to retrieve multiple data values

Expressions can be used to filter choices in lookup_choices

Lookup_choices performs an online lookup to get choices from another form or a CSV file. If you are familiar with using the “search” function to get choices from a csv file then you know that there are some pre-defined criteria for filtering the choices. For example ‘matches’, ‘contains’, ‘in’, etc.

Expressions allow you to be much more flexible in how you select choices. For example in the following video the filter is:

#{votes} > ${min_votes}

So the above filter means that only personnel who have more votes than the minimum are returned. ${} means to use the answer to the question in the form that is being filled in. #{} means to use the answer to the question in the form, or csv file, that is being looked up.

Here is another expression taken from the “Book” example survey in the documentation.

'${time_wanted} > #{from} and ${time_wanted} < #{to} and #{available} = 'OK''

More details on how to define filters can be found here:


  • Improvements to performance of dashboard
  • Automatic refresh of panels in the dashboard is now limited to maps and charts. If you have a table view you will need to manually press refresh in order to update it.
  • Improved tabbing between questions in the online editor. When you tab into a question label it the text will be selected by default so you can just start typing to overwrite it.
  • Add an online tool to look up a surveys identifier so it can be used in lookup_choices, pulldata or other functions that refer to other surveys
  • Support for Ubuntu 20.04 LTS
  • Addition of option to include read only question data in exports. This is now required as people are setting the value of read only questions using calculations. Hence they can contain useful data.
  • Added bulk changes to console. You can now select multiple records and then update all of them to the same values.
  • Improvements to uploading of media from webforms so that more media files can now be uploaded. 100MB+
  • The lookup_labels function also now works in Webforms.

Version 20.08

The last release of the winter of 2020.

New Features

  1. Support for Matrices in WebForms. These can contain mixtures of question types.
  2. The labels of groups are now included in PDF exports
  3. Added a Webform view button on the form management so you no longer need to open the editor in order to try out a new survey in webforms
  4. When selecting columns to show in the console all columns can be selected or deselected using one click
  5. Export of console data in word documents. Data can be formatted as QR codes.
  6. PDF titles are now muli line


A new question type of “begin matrix” is available in order to create matrix structures like that shown below. Further details are in the documentation.

Version 20.05

It looks like we have just got the May release out in time! Enjoy while I get straight into the June release.

New Features

Transcription of Medical Notes

Transcribe Medical can now be used to convert audio into text with special consideration given to medical terms. To use this you will need to add the following parameters to a text question:

  • auto_annotate=yes. This makes the system automatically convert the audio to text otherwise you can use the console to manually do the transcription. You can also use the console to modify an automatic transcription.
  • source=question_name. The source identifies the name of the question that has the audio recording
  • medical=yes. If not set the standard transcription will be used and probably will make a mess of the medical terms.
  • from_lang=en-US. US english is the only language currently supported. Because there is only one valid choice you can omit it.
  • med_type=conversation, or med_type=dictation. If you specify conversation then the transcription will expect more than one voice.

After you submit your survey it can take a few minutes for the transcription to happen. Just keep pressing refresh in the console until you see the results.

Online Editor Support

The online editor now supports the setting of translate and transcribe parameters.

In the online editor select “parameters” as the attribute that you want to edit. Then click on the “edit” button of the text question that is going to hold the transcript. The annotation parameters can be set on the “Annotate” tab of the dialog.


As with the standard transcribe there are fees levied by AWS on the use of transcribe medical. Currently we allow 800 seconds per month free usage. However this free usage has to be enabled for an organisation. Get in touch if you want to try it out or need more than 800 seconds per month.

“In” and “not in” Supported as search functions in WebForms

In the previous release support was added for cascading selects using CSV files in WebForms. These used the ‘matches’ function. For example:

type        name    label     appearance
select_one  country  Country  search(locations)
select_one  city     City     search(locations, 'matches', 'country', ${country})

Now you can also use the ‘in’ function. The list of values is space separated. So the same format as select multiple question. This allows you to do the following:

type            name     label     appearance
select_multiple category  Category  search('list')
select_one      product   Product  search('list', 'in', 'category', ${category})

The products that are in the selected categories will be available for selection in the second question.

“not in” works in a similar way but will only show choices that do not have a match in the list. The other 3 search formats supported by fieldTask are not currently supported in webforms:

  1. contains
  2. startswith
  3. endswith
  4. matches (webforms)
  5. in (webforms)
  6. not in (webforms)

Support for Static Default Images

You can specify the name of an image, from shared resource, as the default value of an image question. For example this could be a diagram on which the user is expected to mark their response.

Other Changes

  • The server is now completely translated into Portuguese
  • You can now set the background colour of buttons and the colour of button text from the web form settings tab on the users page.
  • The server and identity menu options in fieldTask can be disabled on the server
  • When conducting an email mailout you can generate the links and export them to an excel file. This allows you to send the links using other channels to the user. For example from your own email server in a fully customised email message.
  • The guidance hint in surveys can now be enabled or disabled on fieldTask using a server setting.
  • High resolution video capture in fieldTask can now be disabled/enabled from the server
  • Links to embedded files are shown in PDFs
  • The latest mapbox styles are now supported for custom maps
  • Limits have been added to export of survey data for importing into another survey. This overcomes a problem where exporting surveys with tens of thousands of images could cause the server to crash.

Bug Fixes

  • Errors in the task creation rule for a single task group should not now cause the application of rules for other task groups in the same project to also fail.
  • Leading zeros in task durations that previously caused task group rules to fail are now automatically removed.
  • Other miscellaneous crashes reported over the last month

Accountability and Feedback Application Updated

The example application was out of date and did not work. This is because it used the old approach to “oversight” forms. The new approach is that any form can be an oversight form and used to annotate submitted data. My apologies if anyone tried the obsolete example app.

It now works again and I have taken the opportunity to add some artificial intelligence services:

  • Automatic identification of objects in photographs
  • Automatic conversion of an audio recording into text

Another change is that the documentation on example applications has been moved to from its previous location in the “Products” menu.

Version 20.04

New Features

Artificial Intelligence Services

We are continuing to work on adding Artificial Intelligence to the Smap Platform and the following features are a step towards that goal. They add calls to generic AI services provided by AWS for translation, transcription and image recognition.

There is a cost for using these AWS services so by default monthly free limits have been added per organisation.

  • Image Labelling: 100 Images
  • Translation: 5,000 text characters
  • 250 seconds of Audio transcription

Note: These Quantities have been updated as the original post contained the free limits for an entire server. So the new quantity is about 10% of the previous value.

So feel free to use these services and you will get an error message when your monthly limit has been exceeded. If desired, you can then, contact Smap to extend the limit which will incur a monthly fee.

Automatic Translation of Surveys to other languages

The list of supported languages is available in the documentation:

Languages can be translated when the survey is loaded from an XLS template. In this case only blank untranslated text will be modified. You can also initiate auto translation from the translate page of the online editor. In this second case you can specify that the existing language translations be overwritten.

It is important to note that there will be errors in the translations and the performance of AWS translate may vary with language. However this is normal in survey development even with manual translations. Ultimately, during data collector training, problems with the translation get resolved. The online editor translate page is a convenient place to make these last minute adjustments.

The following video shows how to automatically translate a survey in an spreadsheet template.

Automatic Translation of Text Data

Just as you can translate a survey into multiple languages automatically you can now translate text responses in submitted data. This is done by specifying the auto=yes parameter to a question in an oversight form.

The following describes the manual way to translate question responses which can be automated using the “auto” parameter.

Automatic Conversion of Audio into Text

If you collect audio as part of your survey then if the audio is in one of the supported languages you can automatically convert it into text. This then allows you to search through your data for key words that were spoken in audio files. You can add automatic translation of the transcribed text by adding an additional translate question to the oversight form. Details:

Automatic Labelling of Images

This has been available for some time however it has been updated to work like the other services in the background without affecting the speed with which new submissions are processed.

Add “append” for repeat records when updating existing results

If you use a task to update an existing record then any repeating groups in that record will be replaced. The scenario might be that you are doing a follow up interview at a household where there were originally 3 household members. While completing the survey the enumerator might remove one household member, modifying a second and add 2 more. After submission there will be 4 household members with the updated data.

However sometimes you want to append new repeating groups. In particular if you set the parameter “instance_count” to zero then none of the existing repeats would have been downloaded with the task. The enumerator can then only add new repeating records and you would want these to be appended to the existing records.

This can now be done using the parameter: key_policy=append

The hospital example implementation has been updated to include this.

Other Changes

  • Add support for the “required” message in multiple languages
  • Bulk uploading of users, projects and roles from a spreadsheet

Bug Fixes

  • Issue 434. Support the use of apostrophes in text when filtering data on the server.
  • Issue 507. Setting the timezone on tasks exported to a spreadsheet

Version 20.03

This release adds greatly improved support for conducting email surveys. Previously you could conduct mailouts of surveys using the task interface. Now there is a dedicated “Campaign” module. It can be accessed by selecting “Modules” then “Campaign”. Documentation.

Creating a Campaign

A campaign is associated with a survey. You can specify

  • A name for the campaign
  • The subject to be used in sent emails
  • The content of the email
  • A list of email addresses and names

Pre-populate Survey Forms with Data customised for each Recipient

You can include customised data specific to each recipient for any question in the survey.

Manage Contacts

Contacts can be added and edited on a new contacts page. However most commonly a contact will be added indirectly when they are included in a mailout. The contacts page is useful for changing the name of the a contact and also seeing quickly if an email address has unsubscribed.

Campaign API

The API allows you to:

  • Get a list of campaigns
  • Create campaigns
  • Add an email to a campaign

When adding an email you can specify that the URL with the survey should not be sent to the recipient via Smap. The API call will then return the URL to the calling program. This allows you to manage the distribution of survey links in your own application.