so I will talk a little bit about - I will talk about the new delivery from Nettskjema to TSD and Educloud
This is our killing feature as we will come back to this, because this we have been working a LOT with this...
This is designed to scale. We have designed it to scale to like a couple of million submissions a minute.
In TSD we have many files and this is not files, so it scales a lot better.
So to just say again what did we.. what do we have today in...
we could call it the old solution or the xsystem. So today Nettskjema generates a file when you submit it.
We encrypt the file and we send it to your project. There is picked up and decrypted
and it's concatenated together with the other submissions
and then we move the file over to a folder on the durable called CSV files.
This depends on the traffic on Nettskjema but as it is always default enqueued,
it can never be faster than two or three minutes and during the pandemics we had service
collecting hundreds of thousands of submission and it could actually take up to a couple of hours
before people got the data inside.
most researchers can wait a couple of hours for datasets to be ready, but there are other problems we
addressed as tens of millions of files running around in folders inside TSD that do not scale at all
so now we...
we have a what we call "new delivery". So we from the 1st of June we started to use this new delivery
so now with this new delivery, which will run in parallel, when you activate your Nettskjema against TSD or Educloud,
there will be generated a database, like, yeah, for simplicity just call it a database inside TSD or Educloud in your project.
and then that database is ready to get the data from your Netskjema directly.
This means that we can; we are streaming it directly inside and we can allow larger files.
Like now we have 500 megabytes, on attachment we will probably increase the make this one bigger as well.
The data is then stored in the database inside TSD or Educloud and available via an API.
This API is only used by us at the moment so we can use this API to get data off then we have some projects
where we are testing out more functionality.
Leon we'll talk later now how we will use this API to connect to helsenorge.no
now in the beginning we will only take data out which is kind of the same way as we do
with dataloader today but will be instantly available and it will be a new structure.
I will try to show this on a small demo, because this will affect all the users of TSD in a way or another. It could just be nice to just understand this.
We have a kind of a release plan here, after the 1st of June we
said that the new projects that are created in TSD they have this new delivery.
and that means that when they activate their forms, there is created a database and it's sent into the datebase
soon we will allow all the projects to have this for new forms... you will be notified when this is possible.
we will then be able to run in parallel this old and new solution for awhile but
because we cannot just convert the old oldest existing data collection as we have
thousands of Nettskjema delivering hundreds of thousands of answers to TSD every week
so we will only take the new forms in the beginning because we have to create this database on on activation of Nettskjemas
but this Autumn we will go over to let all projects have all their new forms using this new delivery
which will be faster and better, but this old system with the files will continue to run for awhile.
We have not set any end-of-life of that, because we don't see how we can change over existing data collection projects
so there will be then for all project which are created for first of June 2022
there will be these old Nettskjemas that has the data with some delay in one folder structure and the
new ones in another structure. I will show this now on a demo so you can see how this would looks
so if I
then go to my TSD project, yeah, and I go under durable.
my project is an old project that has been now been activated for this new delivery and I've activated one skjema.
we have these old files where we have a CSV files here, three or four we have several that data loader generates on a regular basis as a common job.
I have also attachments, which for technical reasons we create one folder for each Nettskjema even if there is
no attachment. I know this is a little bit hard to navigate through so now I will show
this new structure that we just set up that will be available for the new delivery.
The new delivery will have a folder. Now all new projects have this Nettskjema-data folder
and I can go into this and this is the only new form that I've activated that has a database that
instantly gets data and exports it here via this secure system.
so I go to my phone and here I have my dataset which is an now a CSV file and I have the attachment so we have connected the data set and
attachment in the same folder and we have also been smart enough now to give the
attachments much more robust name so the attachment has now the submission ID and the form ID in the
file name in case of the data being
exported for example
so here you see my files and you can see I've already uploaded a file of 135
megabytes a couple of days ago. If I go to my research form here, I will show you how this work
here you can see my attachments. I don't want to choose this big file, so I will just take a small one now to make it faster
and I can write "demo", and this is a great demo of course, and I can submit my data in to TSD.
It take a little bit longer because it is then sent directly to TSD and you will
get a confirmation when this actually has been delivered so now I'm wondering if I have the demo ghost here
because you can see her at least I have uploaded this photo, this file earlier before this demo at least
It looks like this, hanging a little bit just right now
so there will be then two sets of data, new and old ones.
I think we will give you more information about this when we start to use this
just took a while
yeah this is like in pre-production as you can see but it will be the same in Educloud, but in Educloud
we have another structure but it will be the project area; home area, where we will generate these data
so let me take you back to my presentation here, oh I'm sorry, it's not this one
so for Educloud we will only have this new delivery, we will not use these files for all projects would
only have this new delivery. We will connect this to
Nettskjema and I have some pictures of how it will look now, we will use the same forms
which is today in TSD format will be at TSD or Educloud form, so it will look like
this and when you connect it, it will say that you have to log on to Educloud because you have a
not project specific user, you have a general user. You might have several projects where you
are allowed to post
data to. So you authenticate to Educloud and then we will give you back and you can choose what
project you can connect it to. So if I go to Nettskjema now you can see that we already have made
this available in
when I create a new form and I type Educloud
I can say questionnaire for sensitive data and here this is for TSD or Educloud. I can
connect it as usual and I will just create here and this one is not connected and when I connect it now
we will see that
there is an option here to connect it to Educloud. For those who do not recognize the look
of Nettskjema, I will show this more tomorrow because we then will launch this new version of
Nettskjema, which is more modern. You're not able to click on Educloud today, but hopefully before
summer you will all be able to deliver data directly to Educloud