Nettskjema Integration and Survey API. Speaking of Nettskjema we have worked a lot in the last
year's to improve Nettskjema and our aim towards the consumers of data when TSD started
it was started by a small number of project that was aimed mainly on data processing in this case
they gathered worst vast amount of data they were forcing sensitive data but slowly TSD moved
away not only move the way but
they continue to be there, but TSD give out more and more projects in which researchers wanted
to process their sensitive data but not that much, not on a high level but more on gathering and
obtaining sensitive data so moving, delivering past data faster and more reliable in TSD was dying in
the last couple of years. I will take this moment
to get a little bit reflection of how TSD used to be and how it is now and let's also
take a look at the TSD five years ago when when I started in TSD around five years ago there was an
income that was file encrypted that was encrypted by the user. They were uploaded to an SFTP and then
after one hour probably, this file
will be copied to the project space and this was the case especially problematic with no Nettskjema
because Nettskjema was delivering each answer as a small file to the SFTP that's SFTP was copied to
the
project space after one hour and of course this created even more problem for our non-technical
users which had to aggregate all the other files in a single file we provided a service for that
code the Data Loader but this could take another hour in until these things happens so for the whole
process of Nettskjema it could have taken up to two hours until the data gets their researcher. This
was a situation that was not acceptable to anybody
and we addressed it at least for the power plot several years ago by introduction of the file
API but now we are happy to announce that we want to address and want to give a final enda to this
problem for Nettskjema as well.
So our new Nettskjema now has a new version that does the encryption together with the user. The answer
comes from the user is getting encrypted automatically by Nettskjema. It's getting delivered to
serve API that I'm going to speak to you about today let's get automatic notification that a new
answer to has been delivered to serve API via the RabbitMQ, our notification service agent, and then
it is
automatically loaded inside TSD, and all these things should simultaneous all these things should be done
in couple of seconds. Let's speak a little bit about the survey API and what it does. The survey API
is an API for storing, querying, exporting and exploring data of Nettsjema answers. It is end-to-end
encryption with Nettskjema, so until the data comes from Nettskjema to TSD, it is encrypted completely.
It provides a very faster access to the data as all the data stored on the database of course we
take extra care when we stopped the data in the database so there is aggregation between the data of
different projects and there is no risks involved for the project by having the Central Storage
as well it is more
reliable storage, having more than millions and millions of files on the network on a file system is
not the right way to store the data so moving to a database made more reliable storage as well as
more safe and more sustainable in the long-term
One of the big advantages as well of using the survey API is that each time we get a Nettsjema-answer,
now we can fire a message to process this answer and the processing currently is
involved in an instant loading, so the new Nettskjema forms for the new projects will have a data
loader that will instantly load the
responses to the CSV files from the researchers thus facilitating the researchers with just
surveying the data and know for example when a consent form delivers a consent they can react
straight away and
work with it as well as the new survey API, as it is an API, provides access to the data for
researchers to integrate it with applications there are several projects that already
process automatically data from Nettskjema, and this will make this thing even faster, especially in
the presentation that Patrick did a little bit ago
about food capture, about publication. That when we publish data to patients it will be instantaneous
and here I want to show you a small demo. It is a very very tiny demo it's only 30 seconds. What we
hear and what we see now is a split window in three. On the left you can see a Nettskjema. On the right
you can see a list of a CSV file on the top, with answers, and on the bottom is the Nettskjema
attachments, attachments that are uploaded by Nettskjema.
Notice that the form is very simple it contains your answers but we are also uploading the
file if you can notice the file that we are uploading, it's up to a 500 megabytes, we are being able
to upload what files as well as encrypt them on while uploading and decrypt them instantaneously
inside TSD. I´ll run the demo. The demo is only 30 seconds,
it involves me uploading one, answering the question uploading of files to the TSD
As you can see I answered the file by selecting my file from the disk.
I answered the question.
Maybe a little bit slower. Answer the question poster,
and I hit the send button. When I hit the send button,
the file got updated straight away into the bottom and answer one more question appeared at the
end of the answers in the CSV file on top. This was a live demo, this is practically the speed
that we can obtain in the new survey API. We can deliver answers fast, unreliably to TSD, and
this brings a lot of new opportunities
for integration of Nettskjema data, as I said both in application, in publication solutions, as well as
simply providing a better service to our users.
Should we have more questions now? Well how much data can you now
take from Nettskjema per minute? It is a good question taken in account that in the previous
solution, the delivery of Nettskjema involved getting a lot of tiny, small files and writing them to
the disk which is a very, from a technical point of view, a very slow operation. The true output
through the new survey API to the new survey API makes the whole thing much faster. Operating with databases
is a faster process. We store the file faster we load the data faster and so we can process much more
data in shorter time as well as in the during the last year's we sometimes had technical problems
involving storage, so storage is not a reliable medium for storing answers and moving to the new
survey API, will solve a lot of those. And this new delivery from Nettskjema to TSD
will be a lot of focus on tomorrow but I will also give a demo of the new solution in my foundation
on a Educloud in some minutes, so we will have a more about how this new delivery from Nettskjema
works, because it is awesome and it's scalability and it will change the way we use Nettskjema.
So hang on to this direct stream, to him or
about and see more demos of the new delivery from from Nettskjema.