[Automatic captions by Autotekst using OpenAI Whisper V3. May contain recognition errors.]
I came from FOI.
We are doing analysis and other things at TST.
I would like to talk about the background of what our social project is, and how the project I am in, FIDA,
what that should do.
Then we will go into how we have used ATSD to make use of this.
Then a little further on what we want to explore.
But who am I?
I am an in-lead consultant from ALV.
It was supposed to be one from FOI here, but they have a whole day with other courses today, so I was the one who got to come.
I am a senior at TET, who does a lot of different things, including architecture and development.
I have been doing that since 2013, so I have learned a lot of different things.
Now I have been allowed to be one of those who test TST to use it as a platform.
FOI's social mission is to be prepared and to be able to find information about what happens when there are crisis situations.
To make it accessible for research and such.
And to have health data and that kind of data that is super sensitive.
There are quite a few.
There are 1,300 employees, and they are spread over several locations.
Scalable solutions for awareness should be created.
And to have preparedness situations, such as for example when Corona came, to be able to scale up and do it in a very good way, with data coming in from a lot of sources and large amounts of data, is very important.
To do it in an effective way, and to be able to do large, fast-paced analyses, also requires requirements for the kind of platform you can use.
I looked into what FIDA should do within this.
FOI does a lot of broad work, while FIDA is the integrated data analysis service of FOI.
The goal for it is to have a safe access to the analysis things you need.
You should do large, advanced analyses on sensitive data.
Then you can't use any of these types of solutions.
There are a lot of requirements.
You have to know who has seen which data, why, and what mandate they have to be able to view it.
And that has to be generally in general.
You should also create a simple everyday life for those who do research and do complex analyses, or those who create statistics for health professionals and other experts.
You also create solutions for building competence.
To be able to manage crisis situations.
For example, during the pandemic, the whole FOI was scaled up as an organization.
To be able to manage and onboard many teams is important.
To have it in a good way and have done research beforehand.
The main thing we need to simplify is this.
We need to get data from sources.
We have a lot of sources, a lot of sensitive data.
We need to get that data out to a place where we can analyse it.
And then we need to get it out to someone who can do something useful with the results.
It can be, for example, a municipality supervisor, or a pathologist, or that sort of thing.
An example of full flow is that we have records of vaccines and people who have had positive results on tests and that kind of thing.
Or that you have other infectious diseases that you have a lot of data on in a safe zone that is completely impossible to get to.
Then we will be able to get it out to TSD, run analysis on it, and then we will be able to give it to municipal authorities and visualize it for them.
so that they can know what type of infection is at which places, where we need to take action, for example if it is at a hospital or that type of thing, so that you can quickly come out and take action.
Theoretically, if there had been any type of data, it would have been very tempting to use public cloud.
But since we have sensitive data, and the type of use we have, it is very important that there is someone we can log in and have a proper audit log on, which means knowing who has been there and being completely sure of the identity.
Teste has a very good one where you have login with BankID.
You actually know who people are.
And you have a lot of good logging and control of everything.
You don't have the internet on the normal machine that a normal person has, which also helps a lot.
It is a well-known infrastructure.
It is already known to be used a lot in health data.
We have also seen a lot of potential.
When we have worked together with TSD, we see that we are able to make better solutions together than what we are able to do with a regular cloud provider or infrastructure provider.
Because we don't just get the brain of someone, but we also get some sparring partners that we can sit and get help to make the optimal solutions on how we should do this.
It has been incredibly good for FFI.
What you don't necessarily get out of the box with TSD is infrastructure as a code.
To have shared data across projects and such.
These are things that we build custom in TSD, in collaboration with TSD.
And so you can have...
Those who are not familiar with it, is that you can have analysis projects, where you have one project per team.
But we only want to load data once, and since we want to have data across, which is not what it's supposed to look like, we are dependent on having one real project, which we build all the audit logs into.
In addition, standardization of ETL values is more that you have to script things than you have on a regular public cloud if you are used to it.
Extract, transform, load is ETL.
If you want to make big transformations or move data, you need scripts or some kind of technology that can do that for you.
An example is that one of the data sources we have now is 1.5 billion radars in just the dataset we are going to analyze.
So there is a lot of data.
Crunching it in an effective way, without any developers having to do it, is important.
If we can standardize it as well as we can.
The requirements for running are that you should have daily, secure and automated input and output from the data in TSD.
is something that seems to be a slightly different usage pattern than the others.
As mentioned earlier, it was difficult to get data in and out, but we will try to automate it so that you get it many times a day from each source, in and out data.
So you get delta loads and that kind of thing.
We have a requirement to have fine-grained logging and control of who has done what.
Have as much self-confidence as possible for the analysts, so that they can do their job without having to contact us as much.
And to have container handling, so that you can scale things out in width.
When you do very large analyses, it's not enough with one machine, as I mentioned earlier.
You have to scale it out as well as possible to get it running.
And then it's to be able to scale out on, for example, Docker or that kind of thing.
And to be able to schedule it, so that you don't have to, during corona, you didn't have the opportunity to
say that this would happen so often.
So then researchers stood up at 3 o'clock at night to press the button and go to bed.
It's very, very impractical.
So scheduling is also important.
Logging and scaling as well.
And as a result of this, we have developed this architecture together with TST.
If we divide it up a bit, this is the data flow.
Down to the left, we have our data sources, where we get all of our data within Sikker Zone.
This is a bit technical, but it raises a question.
Then we have scripts to get data in, where ATSD has a finished API that you can use, where we get everything in, that they have secured sharing and things like that.
Then you get it to an import area.
From the import area, you make sure that only individuals have access to it, so you have a lot of control.
And all the data that is
On the way in, when the transport is taken, all the data is encrypted and it is signed, so that we are sure that the data does not get lost.
When we have finally passed the import area, we decrypt the data and put it in a large raw database.
The raw database allows all researchers who have access to the data to start doing access control at the table level and columns, so you can get access to exactly what you need.
Then the researchers can start doing exploratory work, so they can play around with using RStudio, which I set up to be able to
Just coding a bit towards the data sources, and then you can automate them and schedule them out in Bredde, which automatically works, and run Core Review on them.
Then we get these out as results in a new Postgres database, which will automatically be deleted so often.
Then we also encrypt and sign all the data that is transported there.
Then we run it out to the publishing team, which is on another platform, where, for example, municipal supervisors or that type of expert can log in and see the results of analyses.
You can also see the results yourself as a researcher or a statistician within TST.
On the other side, we have the infrastructure to get Hodman images in, where it automatically replicates all images from a registry on the outside every night.
So you can build large containers and do some development on the outside where you have internet.
We have also been able to put up Git repo on the inside.
so that you can have good control over the code.
And to have a task schedule, which you can use to get things to happen automatically and scale the board.
And IAM for using
You can use the same inlogging as you have in TSD to use it on websites that we have within TSD that we build ourselves.
In the same way, you have one password everywhere, so you can have single sign-on across and that kind of thing.
What's next?
For us, it is then to test the whole Schleifa.
We are still in the pilot phase now, so we have the last ROS-analyses and such on speaker, so that we can get things out to the test.
We will also continue to improve the entire automation process and the self-control and self-assessment that we need to set up new teams and onboard people quickly.
And we are going to add several sources.
Today we only have one source that is in so far, which is a large dataset, but we want quite a few.
And we need more users.
Today we have one expert person to test what he has as a requirement, and then we work in more and more universes.
We also want to test HPC, which was presented earlier, to do large analyses.
The researcher who had already started wanted to have about 100 GB of memory on one machine to run his analyses, and that's a lot.
It doesn't work on a normal laptop.
And the last thing is to run the RapidMQ, or the message queue, which is one of the newer features in TSD, I think I understood, which is to be able to get messages out that something has happened, so that you know that, for example, your job is finished running, or that there have been some new warning results that you should go in and check.
Or that the system is down, so you can get that type of notification.
That was it.
Any questions?
Senio Potet, was that you?
You mentioned containers and container images.
Maybe you can explain a little more what it is, for those who are not used to it, and why it is important for you to be able to use it, and what kind of problems does it solve for you?
The easiest way to use a container is to take a whole VM, a whole virtual machine, and then scroll it into one process.
And then you can take that process and parallelize it many times.
It is used a lot to be able to...
In microservice thinking, where you have many specialized microservices, you have many lightweight processes that all have the minimum they need to run.
That is very important for us to be able to spin up.
We have a lot of ready-written libraries to do analyses that we want to have embedded in one process.
which is available to the researchers, but we don't want them to affect each other in any way.
So we want to bring up many different parallels.
And within that, the only difference between the different ones is the one script that the researchers have written that uses the different packages.
So you somehow have a...
The development environment outside, which you replicate inside?
Yes.
We build containers on the outside, like images, so you have a blueprint of what a development environment can look like.
And then you take that and can spin it up exponentially to get it orchestrated.
Historically, from what I have understood, it was a very good idea to start with VMs, where you could set up lightweight data machines and virtual machines, which, when you take it further to containerization, is even more lightweight and can easily spin it up in an incredibly short time.
And then do a lot of good
automatic regulation when things crash, or that you can have high uptime.
Any other questions?
Thank you very much for a very nice presentation.
Up here.
It's incredibly stylish that you can use TSD in this way.
This is part of FOI's social mission.
Is the hypothesis that this will be production, or are we on a test?
We are in the final stages of getting everything approved by ROSAnalyser.
As long as it is approved, the production will continue throughout the summer.
easier analyses in the first place, while we want to do more and more advanced analyses.
FOI does a lot of other analysis work, also in TSD already, but the difference in this project is that you will automatically import and export data from secure sources on a fully automatic path, which is something you rarely do.
Very good.
Good luck.
Thank you.
If this goes in full production and everything is maxed out as you might think it should, and then suddenly a new pandemic is on its way.
So people in the municipalities are collecting a lot of data, and maybe there are hundreds of municipalities that deliver data all the time to this TSD project.
They deliver to the registers, where you have the structured data.
There you have extra personnel who take care that all data has high quality and that everything is correct.
1 billion lines, was it?
It's 1.5 billion on one data source, so that's a lot of data.
And then it comes to the TFT, and there he will run the analysis to see if there are any diseases that look like they react to society in a pandemic-ish way, maybe?
There are a lot of analyses to do to find out, for example, R-counts was one of the analyses we did.
We have had R-counts before at this conference.
Yes, so that type of analysis is done.
The simplest version is to just make statistics, to find out where there is a lot of infection, and to be able to divide it and see trends and that kind of thing.
Analyze what the normal standard is.
What does a normal flu season look like?
What are the outliers?
You can start with anomaly detection to find out what...
When are you going to be careful that something bad happens?
And then the data goes out again, automatically, so that the municipalities themselves can see the trends in their municipalities against other municipalities and so on.
It's pretty cool.
Very exciting, very exciting.
It's fun, because Folkehelse has previously talked to us about this, because they often have very high
It's just a question about the data flow again.
You say that there has been data into the registers.
But are the registers inside TSD, or are they outside TSD, and then the data is locked into TSD?
Outside.
So it's spliced into TSD afterwards, so it's part of the automation?
Yes, correct.
And then it's like, in TSD, you can do an identification or a personal data cleaning, so that it can send out the reports.
Now it's starting to clean up.
In addition, we want to deliver data to both municipalities and to the public.
That's what they call pricking, when you anonymize data.
That's a lot of work to do.
I want to be sure that no one can find out who has been to the doctor or done something, especially when it comes to infectious diseases.
I also come from FOI.
During the pandemic, I have worked a lot with other agencies, and we collect a lot of information from different municipalities.
And we have met a lot of problems with text analysis.
So the last thing we started working on was the test.
So we tried to put in Tintacto and all the questionnaires that would make a product out of it.
So the test is a fantastic system, as far as the doc is concerned.
for example, and POTMAN is very, very, very difficult, because the artificial intelligence comes into the picture.
We can't analyze anything or be effective without having a good infrastructure for artificial intelligence.
And to have it on the test, so you have very good DOC and POTMAN, which makes it a little easier.
and processes that can combine or integrate it.
I think that you can have a big data machine, you can have a lot of power, but without having the components that are easy to install, easy to run.
It's a change, I think.
I agree.
We are trying to standardize the environment for people within FOI, so that they can reuse as much as possible, so that it is easy to use.
For example, as we do with RStudio, you can check in in Git, and it will automatically be picked up and scheduled without you having to worry about containers at all.
I hope we can talk a little bit more later, and then we'll come on board to offer software for both EdoClave and TFT around AI.
Now we've used up your time.
Thank you very much.
Thank you for having me.