Monday 17 June 2013

Final Product

1.     Introductory Product Information

1.1 Name of the product

The intention of the ANDS-funded Marine Virtual laboratory Information System (MARVLIS) is to develop a suite of tools creating value-added products from model and observational data which can be either used by themselves, or used as a ‘plug-in’ library to Marine Virtual laboratory (MARVL), thereby giving MARVL the capability to move from being a research tool for specialists to a tool with a much broader spectrum of users. It includes a clone of Australian Ocean Data Network (AODN),   CSIRO CONNIE tool, Sensor Messaging Gateway and geospatial information portal.

1.2 The primary users of your product

In the Derwent Estuary, the community can now easily access information previously only available on request. The Environment Agency can see ways in which MARVLIS can be used for routine ‘Public Health’ reporting and is assessing options for utilising MARVLIS to speed up, and enhance, future ‘State of the Derwent’ reporting. At present, because of restricted access to the DEP databases data feeds to MARVLIS occurred manually; however, because of the development within MARVLIS of the SMG feeder tools, should these access restictions be removed, then automatic feeds of data to MARVLIS are immediately possible. For MARVL, new user communities can see opportunities from the added library of software tools.

1.3 Problem to solve

For several years the Derwent Estuary has been the subject of local research monitoring of the marine ecosystem in support of the long-term health of the estuary, as impacted upon by historical and ongoing human activity and potential new developments in aquaculture operations (see the Derwent Estuary Program (DEP), http://www.derwentestuary.org.au/, and the CSIRO/UTAS/State Govt program INFORMD, http://www.emg.cmar.csiro.au/www/en/emg/projects/INFORMD.html). These activities have not had a publicly visible on-line vehicle for the data, results and information they have produced. The Derwent Estuary is the focal point for the ‘Early Activity’ of MARVL because of these underlying activities and availablity of models and observations which can readily be used to demonstrate the utility of MARVL. After consultation with the DEP it was agreed that the development of value-added products to provide fast and comprehensive information on the state of the Estuary would be of benefit and could, in the short term, provide the public with information about beach and bay water quality and, in the longer term, offer a mechanism to streamline the 5-year ‘State of the Derwent’ reporting (see http://www.derwentestuary.org.au/state-of-the-derwent-report/).  The tools developed to provide these products could easily be compiled into a library for addition to MARVL.

2.     Instructional Product Information

2.1 What end users said?

The utility of the MARVLIS portal was explored through requested feedback from the DEP partners, who, for the first time, were able to consider the value of seeing their data in the public domain (other than as a table of numbers – the Saturday text recreational water quality report in the local newspaper – or in the State of the Derwent report). The feedback was synthesised by the Director of the DEP and described in the letter of endorsement for the project (see attached document). In summary, the DEP’s response is:

 a) Public health layer:  the layer is useful and easy to navigate, and allows users to track recreational water quality at a given beach or bay over the past six swimming seasons. It also allows users to compare and contrast water quality between beaches and bays across the  system as a whole, and put these results within the context of state and national public health guideleines. The DEP and our council partners would be open to continued use and development of this layer;
     b)  Ecosystem health layer: This layer is useful and easy to navigate, and allows the user to track ambient water quality for a range of parameters at a specific monitoring site throughout the Derwent over a six year period. The sample spatial maps and modeling output would also be useful features for managers. The ability to compare observational and modeled data is also a valuable feature, which offers ‘reality check’ for both the modelled and the observational monitoring program. The DEP and our partners would be open to continued use and development of this layer for some water quality parameters.

2.2 Links to the actual product

2.3 Source code


2.4 User Guide


3.  Product (or Product Components) Re-usability Information

 The Sensor messaging Gateway (SMG) acts as a process engine and broker between the source of data and the publication of that data or any products derived from that data. Formally, this type of system is described as a Message Oriented Middle-ware stream processing engine.

SMG gives the data providers and users the ability to apply complex chains of processing, transforms, alerting, aggregation, filtering, routing, etc. to their streaming data. It allows reuse of these complex modules across any data source entering the engine. It provides a lightweight scripting API for writing sources, processors, timertasks and sinks and is compact enough to run in embedded mode on some sensors and small devices.

SMG communicates over networks / internet using Java Messaging System (JMS) making it easy for users to get data from the field to their systems, users, publication channels and sinks (web sockets may be available in the future too). Importantly it allows automation and reuse of most of the processing and publication tasks that are usually rebuilt for each new data provider and sink.

Connie 2 uses archived currents from oceanographic models and particle tracking techniques to estimate connectivity statistics from user-specified source regions (or to user-specified sink regions). A range of physical and biological behaviors can be specified including vertical migration, horizontal propulsion or swimming (user-specified random or constant velocity), and floatation or surface slick formation (user-specified windage). Ongoing developments include tracking the cumulative exposure of individual particles to environmental conditions (e.g. temperature, salinity, chlorophyll) and integration with other scientific and management tools (e.g. ecosystem models, reserve design tools). Further information can be obtained from http://www.csiro.au/connie2/.  

4. Contextual Product Information

4.1 Licensing of final product

1) Sensor Messaging Gateway (SMG): the feeder scripts to format the DEP data for ingestion into the   
2) SMG is available under CCBY v3.0 AU; the SMG engine is licenced under CSIRO terms and conditions.
3) The CONNIE interface is licenced under GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007; the CONNIE engine is licenced under CSIRO terms and conditions.
4) The AODN geospatial portal and MARVLIS additions are licenced under the GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007
5) The MARVLIS software library of data product tools is available under licence CCBY v3.0 AU.

4.2 Sustainability


1) All the software and data components will be located onto UTAS servers.

2) The MARVLIS portal will be maintained in the public domain for 12 months by eMII/AODN, with support from CRIS and MARVL funding.

3) Negotiations are underway with DEP to establish a mechanism for the delivery of new data – the aspiration is to have the ‘Public health’ data updated weekly, the ‘Ecosystem health’ data at least annually. New data will be added to the database by UTAS staff.
4) If the EPA database holding the DEP data can be placed in the public domain, automated retrieval of data using the SMG can be implemented
5) If new data is added, new metadata records will be created by UTAS staff and uploaded to the RDA

Testing and feedback from users

1. The Application

The intention of the ANDS-funded MARVLIS is to develop a suite of tools creating value-added products from model and observational data which can be either used by themselves or used as a ‘plug-in’ library to MARVL, thereby giving MARVL the capability to move from being a research tool for specialists to a tool with a much broader spectrum of users. The portal can be accessed from http://marvlis.aodn.org.au/marvlis/.

2. Who are the Users?

The users for the application can be marine managers, users of modelling study, researchers and Derwent Estuary Program partners and public users. Marine managers can use the application to run risk management and to access and evaluate the public health and environmental water quality for Derwent Estuary. Users of modelling study and researchers can use this application to learn and run models and to compare model outputs to observational data. Derwent Estuary Program partners in this case are particularly interested in, for the first time, considering the value of seeing their data in the public domain (other than as a table of numbers – the Saturday text recreational water quality report in the local newspaper – or in the State of the Derwent report. Public users now can have easy access to the water quality data and run risk estimate through the application.

3. Testing Methods and Findings

Derwent Estuary Program partners did user acceptance testing and provided valuable feedback. In summary, the DEP’s response is:

 a) Public health layer: the layer is useful and easy to navigate, and allows users to track recreational water quality at a given beach or bay over the past six swimming seasons. It also allows users to compare and contrast water quality between beaches and bays across the system as a whole, and put these results within the context of state and national public health guidelines. The DEP and our council partners would be open to continued use and development of this layer;

 b) Ecosystem health layer: This layer is useful and easy to navigate, and allows the user to track ambient water quality for a range of parameters at a specific monitoring site throughout the Derwent over a six year period. The sample spatial maps and modeling output would also be useful features for managers. The ability to compare observational and modeled data is also a valuable feature, which offers ‘reality check’ for both the modelled and the observational monitoring program. The DEP and our partners would be open to continued use and development of this layer for some water quality parameters.

4. Lessons Learned

1) When engaging with traditionally conservative data users/providers work harder, earlier in the program, to secure appropriate access to the data.
2) Work harder to ensure all parties involved are fully on the same page, and working towards the same deadlines
3) Set an earlier deadline for working systems so that engagement with stakeholders can be more interactive

Sunday 1 July 2012

Glossary

Acronym
Full Name
URL
ANDS Australian National Data Service http://www.ands.org.au
CONNIE http://www.csiro.au/connie2/
CSIRO The Commonwealth Scientific and Industrial Research Organisation http://www.csiro.au
INFORMD http://www.emg.cmar.csiro.au
/www/en/emg/projects
/INFORMD.html
IMOS The Integrated Marine Observing System http://imos.org.au/
NCRIS National Collaborative Research Infrastructure Strategy http://ncris.innovation.gov.au
/Pages/default.aspx
NeCTAR National eResearch Collaboration Tools and Resources http://nectar.org.au/
TPAC The Tasmanian Partnership for Advanced Computing http://www.tpac.org.au
TASMAN The Tasmanian Marine Analysis Network http://www.csiro.au/Outcomes
/Oceans/Understanding
/TasMAN.aspx
UTAS The University of Tasmania http://www.utas.edu.au

Sunday 27 May 2012

KEY TECHNOLOGIES & KEY FEATURES

What is the most important piece of technology being used?

The most important piece of technology being used is the oceanographic model (SHOC), together with particle tracking techniques. The model runs simulations with observational input data to generate data-sets which can be further used and interpreted in MARVLIS. More information about the techniques and models can be found in http://www.csiro.au/connie2/.

What will require the most development effort and why?

The most development effort will be in the SHOC model. The current software only supports archived data and therefore further development is required to make the model accept real-time data.

What features are the most important to gain customer satisfaction and buy-in?

First of all, MARVLIS provides researchers with the infrastructure of MARVL for regional integrated marine studies and embedded tools to aid in environmental assessment. Secondly, the local community will benefit from having a tool to aid management and assessment of the Derwent Estuary, and the national community will benefit from the extensions to MARVL that this project will add. Lastly, the institutions involved will benefit from having tools and products it can use in future projects.


Non-functional requirements

MARVLIS will be built on standards (e.g. Open Geospatial Consortium) and is an open-source project.  All data-sets will be interoperable and metadata will be published to Research Data Australia. There will be different levels of data availability for different groups of users, with the security policy determined by the governing body. Source code will be available through Google Code http://code.google.com/p/marine-virtual-laboratory-information-system/.

Architectural Diagram

Architectural Diagram

MARVLIS will be built from components using a service-oriented architecture. All model outputs will be stored in a THREDDS server and can be accessed through the OPENDAP protocol, with metadata  harvested in GeoNetwork. Map layers will be served via the WMS protocol using GeoServer, from a PostgreSql database. The web server component uses Python and the Django web framework with a PostgreSql/PostGIS database, while the front end is built using Javascript with the OpenLayers mapping framework.

Links for technologies that will be used:

GeoNetwork: http://geonetwork-opensource.org/
Thredds: http://www.unidata.ucar.edu/projects/THREDDS/
PostgreSql: http://www.postgresql.org/
PostGIS: http://postgis.refractions.net/
OPENDAP: http://www.opendap.org/
WMS: http://www.opengeospatial.org/
Python: http://www.python.org/
Django: https://www.djangoproject.com/
OpenLayers: http://openlayers.org/


How the development team works together?

The project will use Agile approach to develop the final product. The project manager will act as scrum master for the development team. Weekly or fortnightly sprint will be used and regularly face to face or video conference meetings will be setup accordingly. Regularly peer to peer code view will be used to ensure all programming codes match the coding standard. Because it is very difficult to get final product users to participate in the development process, the steering committee will act on behalf of end users and is the voice of customers. The steering committee will guide the development process and to ensure the final product is up to the requirements.

Sunday 20 May 2012

PROJECT OUTPUTS & OUR PRIMARY PRODUCT

What are we building?

The main outputs from the MARVLIS project will be:
  • A Library of software routines to compute products from underlying observations and model outputs, as standalone software and as a library integrated into MARVL 
  • Products developed will be tailored to support the management and assessment in the marine environment of a) public health (beach safety and shellfish toxicity) and b) ecosystem health (long-term environmental assessment and consequences of aquaculture management options. 
All software will be documented, and the following documentation will be provided:
  • Descriptions of routines
  • In-line documentation
  • A sample test case
Support? Training?

MARVLIS will have continuous involvement from the Environmental Protection Agency to fine tune products, with the ongoing support from IMOS / CSIRO to ensure that the software evolves and becomes more effective and fit for purpose.

Information about MARVLIS will be disseminated at events to aquaculture operators in the region; and we will be strongly pushing to promote MARVLIS at major conferences locally, nationally and internationally.

Re-use into the future?

Since the foundation of MARVLIS will be a software library built to enhance the MARVL software it will be supported well into at least 2014. The software will be built as a library using standards widely used in the marine and modelling space, so we also expect that the library would also be very transportable for other uses.


How are we going to involve the RDA discovery service?

As well as the software we will also be describing the data and processes in the RDA discovery service. This will also this information to be discoverable, including all the input datasets, the software process, and the derived output datasets.

KEY FACTORS CUSTOMER WILL USE TO JUDGE THE VALUE OF OUR PRODUCT

How is MARVLIS going to be considered successful? & What will MARVLIS's customers be looking for? After some discussion with stake holders in the project, we have determined that the following point need considerable attention to ensure that project will be successful:

  • Customers will need to have the ability to use and download the data products in a user-friendly format. It will be important that the process of data synthesis and reporting are streamlined as much as possible to assist in the user friendliness.
  • Customers will need to ability to compare observational and modelled data with WQO's
  • Customers will expect the observational data to be enhanced with additional contour plots added for a higher resolution output in presentation.
  • Customers should be able to separate and individually select data sources.
  • Customers must be able to interpret any scenario tests, and be able to use then in management options / decision making, and based on the primary focus of the software the customers will expect software derived indicators to assist in monitor the estuary for public health and ecosystem health.

Because the product or software being developed will be designed to be transportable with MARVL it is also expected that it will be easily configured to work in new domains if and when MARVL is used in other estuaries around Australia.



We are also adding some local Derwent data sets as well as output derived data from MARVLIS to the RDA, so we expect that another successful outcome of this project will be that the data and data products will be easily discoverable.

Is it Better?

MARVLIS outcomes are not designed to replace any existing processes, but to enhance, or in some cases streamline what is a manual process. MARVLIS will be seen as an additional tool to better inform policy and decision makers.

Quality and precision?

Because of the nature of the MARVLIS software not all points will be easy to access for quality and precision. It is expected that there will have to be a period of time for the software to prove itself and historical data will have to show a strong trend with the derived data. A proven track record.




Wednesday 2 May 2012

HOW THE PRODUCT WILL MEET OUR USERS NEEDS

If you have read my earlier post about the project, you would have understood that MARVLIS is a project primarily designed to enhance the MARVL project that has been funded through NeCTAR. The Primary outcomes from MARVLIS will be to:

  • generate a data collection for the Derwent publishable in the ANDS Data Commons (Note this is not a data mining exercise, the data are routinely used in environmental assessments, this is more about automatic collation and aggregation utilising MARVL infrastructure).
  • investigate two scenarios which are likely to arise in the management of aquaculture and environmental assessment. These scenarios consider aspects pertinent to both the aquaculture operations and environmental assessment, namely 1) public health, and 2) ecosystem health.
  • create a software library, compatible with MARVL, containing modules to generate data products of value to the scenarios.

So, aside from MARVLIS assisting MARVL by delivering a transportable library of 'functions', MARVLIS will be directly benefiting local business, organisations and community, who use and have a vested interest in the Derwent estuary.

Local shellfish and fish farmers will have access to current water quality and projects future trends, allowing them to be more productive and prepared for changes in the local eco-system. Local government and councils will also have access to additional information, allowing them to be more pro-active and informed, allowing them to better manage the Derwent estuary.

This new data provided by MARVLIS will help in the management of risks in regards to public health, and will lead to quicker closure and opening of farming and swimming beaches. It will also assist in building the local fishing industry through enhanced modelling and 'what-if' scenarios. Allowing new farms and infrastructure to be added for a sustainable future to the local Derwent eco-system.