Final Product

1.     Introductory Product Information

1.1 Name of the product

The intention of the ANDS-funded Marine Virtual laboratory Information System (MARVLIS) is to develop a suite of tools creating value-added products from model and observational data which can be either used by themselves, or used as a ‘plug-in’ library to Marine Virtual laboratory (MARVL), thereby giving MARVL the capability to move from being a research tool for specialists to a tool with a much broader spectrum of users. It includes a clone of Australian Ocean Data Network (AODN),   CSIRO CONNIE tool, Sensor Messaging Gateway and geospatial information portal.

1.2 The primary users of your product

In the Derwent Estuary, the community can now easily access information previously only available on request. The Environment Agency can see ways in which MARVLIS can be used for routine ‘Public Health’ reporting and is assessing options for utilising MARVLIS to speed up, and enhance, future ‘State of the Derwent’ reporting. At present, because of restricted access to the DEP databases data feeds to MARVLIS occurred manually; however, because of the development within MARVLIS of the SMG feeder tools, should these access restictions be removed, then automatic feeds of data to MARVLIS are immediately possible. For MARVL, new user communities can see opportunities from the added library of software tools.

1.3 Problem to solve

For several years the Derwent Estuary has been the subject of local research monitoring of the marine ecosystem in support of the long-term health of the estuary, as impacted upon by historical and ongoing human activity and potential new developments in aquaculture operations (see the Derwent Estuary Program (DEP), http://www.derwentestuary.org.au/, and the CSIRO/UTAS/State Govt program INFORMD, http://www.emg.cmar.csiro.au/www/en/emg/projects/INFORMD.html). These activities have not had a publicly visible on-line vehicle for the data, results and information they have produced. The Derwent Estuary is the focal point for the ‘Early Activity’ of MARVL because of these underlying activities and availablity of models and observations which can readily be used to demonstrate the utility of MARVL. After consultation with the DEP it was agreed that the development of value-added products to provide fast and comprehensive information on the state of the Estuary would be of benefit and could, in the short term, provide the public with information about beach and bay water quality and, in the longer term, offer a mechanism to streamline the 5-year ‘State of the Derwent’ reporting (see http://www.derwentestuary.org.au/state-of-the-derwent-report/).  The tools developed to provide these products could easily be compiled into a library for addition to MARVL.

2.     Instructional Product Information

2.1 What end users said?

The utility of the MARVLIS portal was explored through requested feedback from the DEP partners, who, for the first time, were able to consider the value of seeing their data in the public domain (other than as a table of numbers – the Saturday text recreational water quality report in the local newspaper – or in the State of the Derwent report). The feedback was synthesised by the Director of the DEP and described in the letter of endorsement for the project (see attached document). In summary, the DEP’s response is:

 a) Public health layer:  the layer is useful and easy to navigate, and allows users to track recreational water quality at a given beach or bay over the past six swimming seasons. It also allows users to compare and contrast water quality between beaches and bays across the  system as a whole, and put these results within the context of state and national public health guideleines. The DEP and our council partners would be open to continued use and development of this layer;

     b)  Ecosystem health layer: This layer is useful and easy to navigate, and allows the user to track ambient water quality for a range of parameters at a specific monitoring site throughout the Derwent over a six year period. The sample spatial maps and modeling output would also be useful features for managers. The ability to compare observational and modeled data is also a valuable feature, which offers ‘reality check’ for both the modelled and the observational monitoring program. The DEP and our partners would be open to continued use and development of this layer for some water quality parameters.

2.2 Links to the actual product

http://marvlis.aodn.org.au/marvlis/

2.3 Source code

https://code.google.com/p/marine-virtual-laboratory-information-system/

2.4 User Guide

http://portalhelp.aodn.org.au/Portal2_help/

3.  Product (or Product Components) Re-usability Information

 The Sensor messaging Gateway (SMG) acts as a process engine and broker between the source of data and the publication of that data or any products derived from that data. Formally, this type of system is described as a Message Oriented Middle-ware stream processing engine.

SMG gives the data providers and users the ability to apply complex chains of processing, transforms, alerting, aggregation, filtering, routing, etc. to their streaming data. It allows reuse of these complex modules across any data source entering the engine. It provides a lightweight scripting API for writing sources, processors, timertasks and sinks and is compact enough to run in embedded mode on some sensors and small devices.

SMG communicates over networks / internet using Java Messaging System (JMS) making it easy for users to get data from the field to their systems, users, publication channels and sinks (web sockets may be available in the future too). Importantly it allows automation and reuse of most of the processing and publication tasks that are usually rebuilt for each new data provider and sink.

Connie 2 uses archived currents from oceanographic models and particle tracking techniques to estimate connectivity statistics from user-specified source regions (or to user-specified sink regions). A range of physical and biological behaviors can be specified including vertical migration, horizontal propulsion or swimming (user-specified random or constant velocity), and floatation or surface slick formation (user-specified windage). Ongoing developments include tracking the cumulative exposure of individual particles to environmental conditions (e.g. temperature, salinity, chlorophyll) and integration with other scientific and management tools (e.g. ecosystem models, reserve design tools). Further information can be obtained from http://www.csiro.au/connie2/.  

4. Contextual Product Information

4.1 Licensing of final product

1) Sensor Messaging Gateway (SMG): the feeder scripts to format the DEP data for ingestion into the   

2) SMG is available under CCBY v3.0 AU; the SMG engine is licenced under CSIRO terms and conditions.

3) The CONNIE interface is licenced under GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007; the CONNIE engine is licenced under CSIRO terms and conditions.

4) The AODN geospatial portal and MARVLIS additions are licenced under the GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007

5) The MARVLIS software library of data product tools is available under licence CCBY v3.0 AU.

4.2 Sustainability

1) All the software and data components will be located onto UTAS servers.

2) The MARVLIS portal will be maintained in the public domain for 12 months by eMII/AODN, with support from CRIS and MARVL funding.

3) Negotiations are underway with DEP to establish a mechanism for the delivery of new data – the aspiration is to have the ‘Public health’ data updated weekly, the ‘Ecosystem health’ data at least annually. New data will be added to the database by UTAS staff.

4) If the EPA database holding the DEP data can be placed in the public domain, automated retrieval of data using the SMG can be implemented

5) If new data is added, new metadata records will be created by UTAS staff and uploaded to the RDA

Testing and feedback from users

1. The Application

The intention of the ANDS-funded MARVLIS is to develop a suite of tools creating value-added products from model and observational data which can be either used by themselves or used as a ‘plug-in’ library to MARVL, thereby giving MARVL the capability to move from being a research tool for specialists to a tool with a much broader spectrum of users. The portal can be accessed from http://marvlis.aodn.org.au/marvlis/.

2. Who are the Users?

The users for the application can be marine managers, users of modelling study, researchers and Derwent Estuary Program partners and public users. Marine managers can use the application to run risk management and to access and evaluate the public health and environmental water quality for Derwent Estuary. Users of modelling study and researchers can use this application to learn and run models and to compare model outputs to observational data. Derwent Estuary Program partners in this case are particularly interested in, for the first time, considering the value of seeing their data in the public domain (other than as a table of numbers – the Saturday text recreational water quality report in the local newspaper – or in the State of the Derwent report. Public users now can have easy access to the water quality data and run risk estimate through the application.

3. Testing Methods and Findings

Derwent Estuary Program partners did user acceptance testing and provided valuable feedback. In summary, the DEP’s response is:

 a) Public health layer: the layer is useful and easy to navigate, and allows users to track recreational water quality at a given beach or bay over the past six swimming seasons. It also allows users to compare and contrast water quality between beaches and bays across the system as a whole, and put these results within the context of state and national public health guidelines. The DEP and our council partners would be open to continued use and development of this layer;

 b) Ecosystem health layer: This layer is useful and easy to navigate, and allows the user to track ambient water quality for a range of parameters at a specific monitoring site throughout the Derwent over a six year period. The sample spatial maps and modeling output would also be useful features for managers. The ability to compare observational and modeled data is also a valuable feature, which offers ‘reality check’ for both the modelled and the observational monitoring program. The DEP and our partners would be open to continued use and development of this layer for some water quality parameters.

4. Lessons Learned

1) When engaging with traditionally conservative data users/providers work harder, earlier in the program, to secure appropriate access to the data.
2) Work harder to ensure all parties involved are fully on the same page, and working towards the same deadlines
3) Set an earlier deadline for working systems so that engagement with stakeholders can be more interactive

KEY TECHNOLOGIES & KEY FEATURES

What is the most important piece of technology being used?

The most important piece of technology being used is the oceanographic model (SHOC), together with particle tracking techniques. The model runs simulations with observational input data to generate data-sets which can be further used and interpreted in MARVLIS. More information about the techniques and models can be found in http://www.csiro.au/connie2/.

What will require the most development effort and why?

The most development effort will be in the SHOC model. The current software only supports archived data and therefore further development is required to make the model accept real-time data.

What features are the most important to gain customer satisfaction and buy-in?

First of all, MARVLIS provides researchers with the infrastructure of MARVL for regional integrated marine studies and embedded tools to aid in environmental assessment. Secondly, the local community will benefit from having a tool to aid management and assessment of the Derwent Estuary, and the national community will benefit from the extensions to MARVL that this project will add. Lastly, the institutions involved will benefit from having tools and products it can use in future projects.

Non-functional requirements

MARVLIS will be built on standards (e.g. Open Geospatial Consortium) and is an open-source project.  All data-sets will be interoperable and metadata will be published to Research Data Australia. There will be different levels of data availability for different groups of users, with the security policy determined by the governing body. Source code will be available through Google Code http://code.google.com/p/marine-virtual-laboratory-information-system/.

Architectural Diagram

Architectural Diagram

MARVLIS will be built from components using a service-oriented architecture. All model outputs will be stored in a THREDDS server and can be accessed through the OPENDAP protocol, with metadata  harvested in GeoNetwork. Map layers will be served via the WMS protocol using GeoServer, from a PostgreSql database. The web server component uses Python and the Django web framework with a PostgreSql/PostGIS database, while the front end is built using Javascript with the OpenLayers mapping framework.

Links for technologies that will be used:

GeoNetwork: http://geonetwork-opensource.org/
Thredds: http://www.unidata.ucar.edu/projects/THREDDS/
PostgreSql: http://www.postgresql.org/
PostGIS: http://postgis.refractions.net/
OPENDAP: http://www.opendap.org/
WMS: http://www.opengeospatial.org/
Python: http://www.python.org/
Django: https://www.djangoproject.com/
OpenLayers: http://openlayers.org/

How the development team works together?

The project will use Agile approach to develop the final product. The project manager will act as scrum master for the development team. Weekly or fortnightly sprint will be used and regularly face to face or video conference meetings will be setup accordingly. Regularly peer to peer code view will be used to ensure all programming codes match the coding standard. Because it is very difficult to get final product users to participate in the development process, the steering committee will act on behalf of end users and is the voice of customers. The steering committee will guide the development process and to ensure the final product is up to the requirements.

PROJECT OUTPUTS & OUR PRIMARY PRODUCT

What are we building?

The main outputs from the MARVLIS project will be:

• A Library of software routines to compute products from underlying observations and model outputs, as standalone software and as a library integrated into MARVL 

• Products developed will be tailored to support the management and assessment in the marine environment of a) public health (beach safety and shellfish toxicity) and b) ecosystem health (long-term environmental assessment and consequences of aquaculture management options. 

All software will be documented, and the following documentation will be provided:

• Descriptions of routines
• In-line documentation
• A sample test case

Support? Training?

MARVLIS will have continuous involvement from the Environmental Protection Agency to fine tune products, with the ongoing support from IMOS / CSIRO to ensure that the software evolves and becomes more effective and fit for purpose.

Information about MARVLIS will be disseminated at events to aquaculture operators in the region; and we will be strongly pushing to promote MARVLIS at major conferences locally, nationally and internationally.

Re-use into the future?

Since the foundation of MARVLIS will be a software library built to enhance the MARVL software it will be supported well into at least 2014. The software will be built as a library using standards widely used in the marine and modelling space, so we also expect that the library would also be very transportable for other uses.


How are we going to involve the RDA discovery service?

As well as the software we will also be describing the data and processes in the RDA discovery service. This will also this information to be discoverable, including all the input datasets, the software process, and the derived output datasets.

KEY FACTORS CUSTOMER WILL USE TO JUDGE THE VALUE OF OUR PRODUCT

How is MARVLIS going to be considered successful? & What will MARVLIS's customers be looking for? After some discussion with stake holders in the project, we have determined that the following point need considerable attention to ensure that project will be successful:

• Customers will need to have the ability to use and download the data products in a user-friendly format. It will be important that the process of data synthesis and reporting are streamlined as much as possible to assist in the user friendliness.
• Customers will need to ability to compare observational and modelled data with WQO's
• Customers will expect the observational data to be enhanced with additional contour plots added for a higher resolution output in presentation.
• Customers should be able to separate and individually select data sources.
• Customers must be able to interpret any scenario tests, and be able to use then in management options / decision making, and based on the primary focus of the software the customers will expect software derived indicators to assist in monitor the estuary for public health and ecosystem health.

Because the product or software being developed will be designed to be transportable with MARVL it is also expected that it will be easily configured to work in new domains if and when MARVL is used in other estuaries around Australia.


We are also adding some local Derwent data sets as well as output derived data from MARVLIS to the RDA, so we expect that another successful outcome of this project will be that the data and data products will be easily discoverable.

Is it Better?

MARVLIS outcomes are not designed to replace any existing processes, but to enhance, or in some cases streamline what is a manual process. MARVLIS will be seen as an additional tool to better inform policy and decision makers.

Quality and precision?

Because of the nature of the MARVLIS software not all points will be easy to access for quality and precision. It is expected that there will have to be a period of time for the software to prove itself and historical data will have to show a strong trend with the derived data. A proven track record.

HOW THE PRODUCT WILL MEET OUR USERS NEEDS

If you have read my earlier post about the project, you would have understood that MARVLIS is a project primarily designed to enhance the MARVL project that has been funded through NeCTAR. The Primary outcomes from MARVLIS will be to:

• generate a data collection for the Derwent publishable in the ANDS Data Commons (Note this is not a data mining exercise, the data are routinely used in environmental assessments, this is more about automatic collation and aggregation utilising MARVL infrastructure).

• investigate two scenarios which are likely to arise in the management of aquaculture and environmental assessment. These scenarios consider aspects pertinent to both the aquaculture operations and environmental assessment, namely 1) public health, and 2) ecosystem health.

• create a software library, compatible with MARVL, containing modules to generate data products of value to the scenarios.

So, aside from MARVLIS assisting MARVL by delivering a transportable library of 'functions', MARVLIS will be directly benefiting local business, organisations and community, who use and have a vested interest in the Derwent estuary.

Local shellfish and fish farmers will have access to current water quality and projects future trends, allowing them to be more productive and prepared for changes in the local eco-system. Local government and councils will also have access to additional information, allowing them to be more pro-active and informed, allowing them to better manage the Derwent estuary.

This new data provided by MARVLIS will help in the management of risks in regards to public health, and will lead to quicker closure and opening of farming and swimming beaches. It will also assist in building the local fishing industry through enhanced modelling and 'what-if' scenarios. Allowing new farms and infrastructure to be added for a sustainable future to the local Derwent eco-system.

THE PRODUCT TEAM

"So we know what the product is about, but who will be working on this project?"

Brendan Davey,
Project Manager

Brendan has worked in the IT space for 17 years, having experience in IT service delivery, IT administration and management as well as software development in both the private and public sector. Brendan is currently involved in delivering projects funded via ANDS and NeCTAR and is the software development manager for TPAC.

Scott Condie,
Lead Software Engineer (CSIRO)

Scott Condie leads a research group that specialises in ecosystem modelling and integrated marine management. His research interests include modelling physical-ecological interactions in marine and coastal environments and development of decision support tools. Scott has been extensively involved with the development of CONNIE2.

Mark Hepburn,
Software Engineer (CSIRO)

Mark is a generalist software engineer who has worked in the CSIRO for 5 years.  He has been involved in robust data delivery and processing frameworks, bioinformatics, model coupling, visualisation, and tool development.  Most recently he has developed a number of rich web applications for visualising complex scientific information in an intuitive and explorable manner.  He holds a Bachelor degree in Electronic Engineering and a PhD in Computer Science, both from the University of Tasmania.

Ming Fu,
Software Engineer (TPAC)

Ming Fu holds a Bachelor and Master degree in Computing, and has worked in web-based application development for over 5 years. He has developed more than 10 commercially deployed web applications with experience ranges through online gaming, financial services, human resource management, education and tourism. He has web development experience from both front-end and back-end using various web technologies. Technologies familiar with are Java, html, css, javascript, jpa, hibernate, jquery, struts2, spring, mysql, postgresql, ant, maven and tomcat.

Since 2010, he has been employed as a software engineer at the Australian Research Collaboration Service and Tasmanian Partnership for Advanced Computing, working on various web application projects in e-research area and has been involved in several ANDS funded projects. He has a strong interest in software development and is keen to provide software solutions in e-research area.

In his spare time, he enjoys learning new technologies and writing software for fun.

THE PROJECT DESCRIPTION

Hi,

Welcome to our Blog about MARVLIS. "What is MARVLIS?" I hear you ask, to answer this question let me explain the background on how MARVLIS was conceptualised:

The Integrated Marine Observing System (IMOS), a component of the National Collaborative Research Infrastructure Strategy (NCRIS), hosted by the University of Tasmania (UTAS), was invited by the National eResearch Collaborative Tools and Resources (NeCTAR) program to establish an ‘early activity’ Virtual Laboratory. The proposal put forward by UTAS is for a Marine Virtual Laboratory (MARVL) bringing together IMOS with the CSIRO Wealth from Oceans ocean modelling group to develop a relocatable framework for synthesising observations and models in any marine domain.

The intention of MARVL is to provide a generic tool for significantly reducing the time needed to initiate scientific studies or develop management scenarios, establishing a national infrastructure for integration of modelling and observational data. A case study demonstrator of MARVL is to be the Derwent Estuary region of South East Tasmania because of the ongoing existing modelling and observation program being conducted there.

The Derwent Estuary, including the d’Entrecasteaux Channel, in SE Tasmania is a temperate estuary with strong ocean interaction and has direct application to management of multiple uses, including fisheries and aquaculture. A number of environmental issues affect the estuary:

·       Heavy metal contamination of water, sediments and seafood
·       Loss of estuarine habitat and species
·       Introduced marine pests and weeds
·       Altered river flow regimes and blocked fish migration routes
·       Elevated levels of nutrients and organic matter, and low dissolved oxygen levels.

Understanding these interactions and potential conflicts has been the subject of a joint UTAS / CSIRO / State Government project called INFORMD which has attempted to integrate field studies, the TASMAN sensor network (www.csiro.au/TASMAN) and a coupled hydrodynamic-biogeochemical modelling system to explore the fundamental processes impacting on the circulation and water quality of the estuary.

The aspiration is that datasets generated by these activities will be used to inform management options and provide risk assessments (delivered through tools such as CONNIE, http://www.csiro.au/connie2/).

This project arose though discussions with ANDS on the prospects of constructing software (an Information System - MARVLIS) to demonstrate how scientific outputs (observations and models produced by a system like MARVL) can be enhanced to provide value-added information to support the management and risk assessment of the activities in this regional water body.

Aims and Objectives

The aims of this project (MARVLIS) are to enhance MARVL by the creation of additional software to:

a)     Develop value-added data products to enhance existing marine management practices, and

b)     Provide at least 2 demonstrations of these enhancements.

The specific objectives of MARVLIS are to:

(i)     Generate a data collection for the Derwent publishable in the ANDS Data Commons (Note this is not a data mining exercise, the data are routinely used in environmental assessments, this is more about automatic collation and aggregation utilising MARVL infrastructure).

(ii)   Investigate two scenarios which are likely to arise in the management of aquaculture and environmental assessment. These scenarios consider aspects pertinent to both the aquaculture operations and environmental assessment, namely 1) public health, and 2) ecosystem health.

(iii)  Create a software library, compatible with MARVL, containing modules to generate data products of value to the scenarios.

Outputs and Outcomes

Project outputs

(i)     A library of software tools compatible with MARVL

(ii)   Software tools to generate value-added products applicable to the selected scenarios

(iii)  Products, derived from underlying scientific data, supporting the scenarios. For public health this will include the impacts of ‘events’ on water quality which may influence the temporary closure of beaches or shellfish farms; for ecosystem health this will include ‘what-if’ management scenarios examining the impact of moving or expanding the scale of fish farms; the impacts and optimisation of coastal zone development, and evaluating products for increasing the cost-effectiveness of Environmental Impact Assessment.

Project outcomes

(i)    Enhancement of MARVL to deliver regional management support tools

(ii)   New tools to aid public health and ecosystem health assessments

(iii)  Improved coordination between aquaculture management, environmental assessment and scientific studies in the Derwent Estuary region

(iv)  Improved working practices for aquaculture operation and environmental assessment in the Derwent Estuary region, with implications nation-wide.

---