Semaphore

Semaphore is a tool for the (Semi-) automatic analysis of Australian ecosystem dynamics.


Leave a comment

Semaphore Project Description

Semaphore is a tool for the (Semi-) automatic analysis of Australian ecosystem dynamics.

Ecosystems are in a constant state of change. They are complex dynamic systems that respond and adapt to extreme events and stimuli such as droughts, cold winters or human activity.

Understanding the dynamics of the ecosystem (how it responds to disturbance) can aid environmental management decision and policy making.

Scientists study ecosystem dynamics by monitoring change in carbon cycles and the flow of nutrients in soil and vegetation. They build models of how nutrients and energy flow in ecosystems, using empirical data that they can later use to simulate future changes and predict their effects.

Predictive modeling can be extremely labour-intensive however.  Raw data needs preparation to be used as input in modeling software. Predicted results are then painstakingly compared to historical data to calibrate the software to local conditions.

Semaphore is an attempt to transform this manual activity into a semi- or fully-automated process. Automation reduces the possibility of human error when processing and analysing the data.  It also helps provide provenance information to trace and validate predictions and model calibrations.

Continue reading


Leave a comment

Semaphore: The video

And now in glorious technicolour, Semaphore: the video!

Thanks to the researchers and developers who took time out to feature in this clip. We hope that it will demonstrate just what Semaphore does and its value to Soil science and environmental modelling in Australia.


Leave a comment

Final Product Post

Introducing Semaphore

The AP29 project (Semaphore) has reached the end of its development and we are happy to announce that Semaphore tools are available for the general public.

Semaphore is a web application which runs biogeochemical modelling tools online and also has a Microsoft Excel addin to help importing the modelling output. For those who want to use Kepler Scientific Workflow to run the biogeochemical models, a Kepler workflow example bundled with custom actors is also provided.

The primary users of Semaphore tools will be scientists studying greenhouse gas emissions. Particularly, those scientists focusing on emissions from agriculture and using various greenhouse gas modelling tools to predict emissions and test assumptions. Previously, users needed to have a local copy of the modelling software on their pc which was time consuming and complicated to install and set up. Calibrating the model is also complicated and needs to be repeated many times, which makes keeping track of the data and outputs difficult. By using the Semaphore web application, a user can easily create, manage, and run their experiments in the cloud without having to download or install the modelling tools. All the associated input and output data for each experiment run are made persistent in the cloud and allow them to be easily tracked.

To validate the modelling result, Scientists use Microsoft Excel to quickly transform data, generate charts, and compare results visually. However, getting the modelling output to Excel spreadsheets is also a tedious manual process involving a lot of clicks and copy paste repeats.  To aid in this process Semaphore has an Excel addin which means model output can be imported with a single click, again streamlining a complicated process.

Using Semaphore

Semaphore was developed in consultation with Scientists who use climate modelling software on a regular basis.
Our star Semaphore testers are Dr Martin Labadz and Dr David Rowlings who are both soil scientists working in the N2O network. You can see interviews with them and other scientists using climate modelling tools in our Demonstration of Value video coming soon to the Semaphore blog.

The Semaphore software itself is easy to access and use.

New users should go to the front page of the Semaphore website where you can request log in credentials. Users with a password can then log in and begin using the software straight away. Log in takes the user to a list of their previous experiments and the option to begin a new model run

Screen shot of Semaphore experiments list.

When a specific experiment is selected all input files, calibration notes and output files are visible on a single page. Experiment page screen shot.

Century and Daycent are biogeochemical model that simulates fluxes of Carbon (C) and Nitrogen (N) among the atmosphere, vegetation, and soil. Currently, Semaphore supports running the Century and Daycent modelling tools online,  however, the Semaphore system is designed to be generic and modular, which allows more modelling tools that are not limited to a particular domain, to be added by developers. Any program that can be run from command line interface, accept input file, and produce output file can be plugged in to the system in a relatively easy manner. Theoretically, Semaphore can be re-used for any discipline that involves computer simulation and needs to manage and run the experiments online.

Inside Semaphore

Component-wise, several Node.js modules have been developed as parts of this project. These modules can be re-used in other projects.

  • jsx : Various helper utilities for strings, arrays, file system access, logger, etc.
  • openstack : A library to simplify access to Openstack SWIFT cloud file storage.
  • reqis-jobq : A general purpose job priority queue backed by redis for data persistency.
  • modex : A library that extend express web application framework to provide pluggable modularity of application components.

All source for Semaphore code is publicly available at https://github.com/QUT-IFE-eResearch/semaphore and https://github.com/QUT-IFE-eResearch/semaphore_wms

For more end-user and technical documentation, please refer to the project wiki at GitHub: https://github.com/QUT-IFE-eResearch/semaphore/wiki

All code is licensed under the GNU Lesser General Public License (LGPG) version 3. All documentation is licensed under Creative Commons Attribution-Share Alike. The code and documentation are publicly available from the GitHub repository[link here] and can be reused by others.

Semaphore in the future

Semaphore as a product is still at its infancy and under active development. The team at the IFE HEEM and the N2O Network soil scientists in particular have plans for adding further modelling tools and improving the discoverability and shareability of data outside of their network.

The sustainability of the product has been considered throughout the project and the software has been designed to maximize future maintainability. Key steps taken include:

  • Making sure the code is open source so that other groups can contribute code back to the project.
  • Using GitHub makes it easy for changes from other groups to easily be incorporated back into the main code base.
  • Choosing JavaScript, the de facto programming language of the web, for developing both the front-end and back-end of the system.
  • Writing extensive documentation to explain the design and modules of the system.

Deployment and Testing: Semaphore is go!

Leave a comment

Currently, scientists studying greenhouse gas emissions focusing specifically on emissions from agriculture, use various greenhouse gas modelling tools to predict emissions and test assumptions.  Scientists in the field collect and record physical measurements of emissions from the ground based on various different farming practices and use those measurements to calibrate the models. Once the models are calibrated they can use them to make predictions based on different scenarios, and they can validate the model predictions by carrying out experiments in the field and recording measurements.

Semaphore is a web based tool that simplifies environmental modelling processes, making a complex and laborious task faster and more straightforward.

We have designed Semaphore so that researchers can upload files, run multiple model simulations and control versions of input and output files, all in the cloud.

Semaphore UI screenshot

A screenshot of the Semaphore experiment screen

 

Users can begin a new experiment and select to use Century or DayCent models (more modelling tools will be added in the future).

Semaphore new experiment set up

Creating a new experiment.

 

Users can upload previously formatted data or use our Excel plug in to upload data straight out of Excel in a Century or DayCent format ready to run.Screenshot of the Semaphore Excel plug-in

The Semaphore excel plug-in.

 When a model is run, all the files associated with the model are stored, along with any associated model outputs in the cloud, for you to download, run again or modify.

Screenshot of model runs.

Model experiment runs.

 Records of the model results and parameters are recorded in the run history so researchers can easily see the calibration changes for each model run.

Screenshot of Semaphore model runs

Run history documentation

 

Semaphore users and testers

Our testers for Semaphore are Dr Martin Labadz and Dr David Rowlings.

We have chosen them as representative of the target audience for Semaphore, which is Scientists studying greenhouse gas emissions in Australian soils.

The data that is being used to test Semaphore is from the National Agricultural Nitrous Oxide Emissions Research in Australia program (www.n2o.net.au). Dr Labadz and Dr Rowlings are both researchers in the N2O network and have practical experience using the Century and DayCent modelling software.

Testing methods and findings

Early on in this project we interviewed the researchers and observed them in the modelling process to understand their workflows and identify where our technology could help streamline the activity and yield more effective and easily shared outputs.

Throughout the development of Semaphore we continued to consult with our steering committee by email and regular committee meetings. Following their advice we arranged meetings with researchers using a variety of modelling tools and workflows so we were able to develop a picture of the best features for our product.

We also had continued contact with our testers and there was a demonstration at each phase of development. There was also usability testing by the researchers in the final phase of the project.

Consultation

In the most recent demonstrations and usability testing sessions the researchers were very positive about Semaphore and its use for their environmental modelling work.

Points made were the clean look of the user interface, the ‘safety net’ feature of running the models and storing data on the cloud and also the ease of tracing calibration changes to the model. The Excel plug-in was also very well received.

Regarding future development of the product the testers suggested that a Graphical UI would be a great addition and some data cleaning tools would also be useful.

Soon a Demonstration of Value video will be available here to illustrate the reception of the product by the researchers and the development team.

This gallery contains 5 photos


Leave a comment

Interface Design

Semaphore is fast approaching completion and with a week to go there are many things happening.

The most visible result of our activity is in the new version of the Semaphore User Interface.

Alvin has been working with our designer Steph on the look and feel of the web pages  and as you can see in this screen shot, the result is a clean and intuitive design sporting our new logo and colours.Semaphore UI new design screen shot

 


Leave a comment

Semaphore visuals

The user interface for Semaphore is taking shape and this week we have visuals from the first stages of the web app and the excel plug-in.

The web application UI  has been designed with a simple singe screen where a researcher can log in and create a new modelling project, edit project details and keep a record workflows. When a model is run, all the files associated with the model are stored, along with any associated model outputs in the cloud, for users to download, run again or modify.Semaphore UI screenshot

Alvin has also been working on the Semaphore Excel plug-in which allows researchers to upload data straight out of excel into the semaphore web app.

Semaphore excel plug in Screenshot

Fine tuning and testing is now underway and in the next update we will discuss the test process and the demonstration of value for Semaphore.


Leave a comment

Semaphore’s Value

To follow on from the previous post about the future of Semaphore, we take a look at the week’s developments with regard to our next project deliverable: Demonstration of Value.

This week saw a meeting between members of the Semaphore committee and co-ordinators of the N2O network.

The N2O Network is a collaborative research program, co-ordinated by the Institute for Future Environments.  It was established to study nitrous oxide emissions and to support tools that will help researchers, growers and policy makers in reducing these emissions in Australian agricultural soils.

The Semaphore tool we are developing has value to such a project and it is hoped that a collaboration will allow the life of the Semaphore project to be extended.

The idea for future development in conjunction with the N2O project is to standardize input parameters for the different models as well as adding additional models to the automation tool.

Work continues with  a focus now on demonstrating the value of the Semaphore tool.


1 Comment

Semaphore Sustainability

These past weeks have involved discussions on the direction in which Semaphore is heading: towards the end of the project and beyond.

Here is a basic outline of notes that developer Alvin and business analyst Jason have written up from their discussions on what the vision of Semaphore has become, and where it could head.  All of these iterations are suggestions based on the view that the Semaphore software tool will be used,  and have most value to the intended customer in a cloud based environment.

The first step is having a basic “project” web application where a user can log in and create a new project; where they can list or search for existing projects; provide input files (either through creation, editing or uploading); run a model; and download the output file.  It is also envisioned that the input file editing capabilities will be through a very simple text editor.

There will be online processing of output files to produce visualisation; and a better editor for the output files.   This editor will allow for example forms to create or edit input parameters, and perhaps abstract and rationalise the input files; export weather data from Bureau of Meteorology (BOM) etc.  Enhancements are hoped to include supporting metadata, RIF-CS format etc.

It is also hoped that by June there will be  flexible REST data transformation services based on Javascript.  Through this users could create a new service by either writing new scripts or combining existing services.  This will provide more flexibility to input/output processing and visualisation.

The sustainability of Semaphore lies in delivering a cloud-based tool.  This opens up the possibility of further development, hopefully towards a tool that allows ease of use and a means of sharing input and output data with a wider community that subsequently improves modelling output.