[Automatic captions by Autotekst using OpenAI Whisper V3. May contain recognition errors.] So hello, everyone.
My name is Arash Ahmadi.
I work at the Science Center as a head of data science analytics group.
And today I'm going to talk about one of the projects in which we used EduCloud, which is, in fact, an exploratory data analysis tool that is built inside EduCloud and is called Insights Oslofjord.
And here is my plan for the next 20 minutes.
I briefly introduce the science and data science analytics group.
And then I talk about this Frisco SoFjord project, which is a parent project to Insight SoFjord.
And then I explain to you how we use EduCloud in the context of Frisco SoFjord.
Then I will describe the data pipeline and the server architecture behind Insight also fueled.
And since this is our Shiny application, I briefly try to explain to you what our Shiny applications are and how they function.
And in the end, if you have time, I briefly showcase the app and some of its features.
So D-Science is a center for computational and data science operating under the faculty of mathematical and natural sciences.
And we have four main mandates.
One is sustaining long-term research, initiating collaboration across disciplines, industrial and public sectors, integration of data science in research projects, supporting education and development of skills.
If you are curious about these things, please visit this web page and read more.
And if you are wondering about something, please feel free to reach out to us at this address, contact at thescience.uio.no.
But our group inside the data science analytics group is mainly focused on this mandate, which is integration of data science in research projects.
And we do so through two main themes.
One is these targeted courses and workshops that we hold on commonly encountered challenges by researchers at UIO.
And that can include things from data management to software development and data visualization.
And the other theme is direct involvement in research projects, where we actually receive applications from research groups within Faculty of Mathematical and National Science and this science partner where actually they describe
a current computational and data science challenge that they're facing, and we try to get involved and come up with solutions.
And the projects that I'm going to talk about is actually one of those cases.
And we have a very close collaboration with IT departments, mainly FFU and Nike.
And again, if you want to know more about our activities, you can find more information at this web address or
inquire anything if you want.
So that was about the science.
Now, a few words about this free source of your project.
This is a project initially owned and managed by Institutes of Marine Research.
It has multiple partners, including
Konzberg Maritime, Inspiria Science Center and D-Science and many more partners.
And it has three overall goals.
To generate new knowledge and basis for future management of the fjord.
Test and verify new technologies for monitoring the fjord.
Communicating to the policy makers, general public and especially younger generations.
And data collection and analysis is central in this project.
And the data comes from a variety of different sources, one of them being this website called Dugnaut voor Havre, which is a place where people go and register their findings when it comes to biology or ecology around and inside Oslofjord.
And they can actually tag it to a specific geolocation.
And this way, this website has grown to be a massive data center for direct observation of people around Oslofjord.
Another source of data is a boat called Rangfjell.
This is just an exemplary boat.
But the boat takes students from different high schools in Oslo and they actually perform some excursion into Oslofjord
have some direct observations into the fjord.
And at the same time, this boat is equipped with some sensors that collects the data from the water column.
And this is one source of data.
Another source or category of data is actually coming from these ferry boxes that are installed on the cruise, or maybe I should say under the cruise ships or the ferries that travel regularly around Oslo fjord.
And they collect the data from the water.
Another source of data is this underwater observatory called Lander that is actually laying somewhere at the bottom of the ocean and collects data from the water column above it and it takes some imaging data from its surrounding.
So now that we have these different sources of data, the idea is to actually
use this data in the context of research and education.
And here at DeScience and at UIO, we have the capacities to do so because of the expertise and infrastructure, both in data and computational sciences and in domain areas.
And the expertise basically comes from a network of researchers, postdocs, PhD students, and master's students that perform research on the relevant areas.
platform or the infrastructure here is actually, it should be a set of platform for collaborative data warehousing and analysis that is equipped with HPC.
And for our application in this case is EduCloud.
We use EduCloud as a main platform for this project.
And the way that is going to function ideally is that we create metadata for each data source and we combine them into a universal convention.
And upon interaction with these metadata, researchers can actually form hypotheses.
And if they end up needing any of these data sources, we can pull it into the cloud and give multiple access to different research groups.
And this has one immediate benefit, and that's that we don't need to anymore transfer data from here to there and keep multiple
duplicates of the same data in different place.
The data is centralized and everyone has access to it and can analyze it.
The other benefit is that there's a HPC infrastructure in the same platform where the data lays.
That's again another important thing.
If you have a massive data, you don't need to move around and put it where the HPC is.
That's already there and you can plug it into Fox and analyze the data.
So this makes the data analysis very efficient and hopefully in the long run it feeds into the productivity of researchers when it comes to performing the research and creating educational material from this.
In parallel, we are creating an interactive analysis tool that allows for exploration of these data sources somehow deeper.
We create, we preprocess the data, analyze
and create some visualizations and people can use this app and actually visualize and explore into the data.
The primary users of this applications are students and teachers in high schools around Oslo.
I will talk about this in the next slide, but another group of users for this application can be researchers actually working in this area.
They can actually get into this application and explore the data in an interactive way.
And if they need the raw data for further analysis, that's available on gcloud, and they can have access to it.
And once they come up with some findings and their research is done, they can implement the results of their finding into the same application and sort of enrich the domain area of this application.
And in the end, if such applications are mature enough and there are concrete and well-defined findings that can benefit wider audiences,
Such applications are very efficient and effective tools to communicate things to the general public.
So this was an overview of how we use EduCloud in the context of this big Frisk Osofjord project.
Now I'm going to narrow it down to this application that I was talking about, that is called Insight Osofjord.
The data for this application comes from this boat called Runfield.
That's the real boat.
And the data is actually collected in a collaboration between Konzberg Discovery and Inspirio Science Center.
And the way it functions is that they take students from different high schools in Oslo to daily excursion in Oslofjord.
And they actually collect some biological samples from the fjord.
And with the help of biologists on board, they try to identify what they are.
And they register it on a Lugnot for Haba website.
And this becomes one source of the data.
And the boat is equipped with some sensors that collects information on things like weather conditions around the routes that the boat travels.
And it has an echo sounder that collects the zonal data from the water column under the boat.
And the initial motivation for this application was to actually create an interactive tool for those students to be able to look into their data and interact with it after they're done with their trips.
But the user population behind this application can easily go beyond this, teachers and students in the high school, as I explained to you.
So now I'm going to talk about the data pipeline and the server architecture behind Insight Osofjord and basically
explain to you how we get from this raw data to this interactive analysis tool.
The data initially lays in the data provider's repository.
And they have different formats.
And they actually are in different locations.
And they are sort of sporadically uploaded.
And there's no clear schedule for uploads of data.
We want all those data to be collected in EduCloud project repository.
And since that data is actually uploaded not synchronized in a way, we need to use a persistent server that constantly checks for uploads of those data.
And that's done through the APIs that is run by this persistent server.
And as soon as the data comes into existence in the
data provider's repository, it's pulled into EduCloud.
And once the data is completed, it adds to around six gigabytes per day.
And this triggers a pre-processing step that basically transforms these machine-native binary data into structured tabular data in a tidy format for further analysis.
And this is around 266 million rows of data.
And this is beyond what normal computers can handle easily.
That's why we actually form most of our batch analysis in Fox.
And in those analysis, we extract the relevant parameters that we are interested in.
And we integrate different data types into a time series.
And then we aggregate those time series over those selected parameters.
This way, we clean and tidy the data.
And this is reduced to around 30 kilobytes per day.
From then on, we don't need the Fox anymore.
We actually have another persistent server.
And we push this tidy, cleaned up, or sort of cooked up data for visualization into second server.
And there we create this interactive visualizations using Shiny.
And then using another software called Shiny Server, we push this output into a local host.
And in the next step, we use NGINX to port the content of this local host into this web address that becomes this application that people can interact with over the web.
And it can be found in this address, insect-ossofjord.egcloud.no.
And this is basically the pipeline behind the app, and it functions automatically.
It has a life of its own.
We try to leave it alone as much as possible, but sometimes we need to get involved, and that's where the data, there are some inconsistencies in the data.
And that's important when you develop an automatic pipeline.
You need to have consistent data structure.
And if there is any change in the structure, that breaks down the whole pipeline.
So that's the only thing that we need to actually from time to time get involved and make the app immune to those errors.
Otherwise, it functions automatically.
So since I have been talking about or mentioning Shiny a few times, I thought I can use a minute to explain to you what they are just in case you haven't heard of them.
So Shiny is an R package that makes it easy to build interactive web apps straight from R and Python.
It has two main components, a user interface and a server.
In the user interface, the user has the possibility to give an input and see the output.
And in the server part of the app, the output is being actually built.
And the key point about it, about the Shiny applications, is that in the process of building this application, this output, which can be in this case a plot, in the process of building it, you can make use of the same value that the user has given.
And then you can channel the same plot that has been created for users to interact with.
So this is the interactive concept behind Shiny.
It sort of allows the user to define what they're going to see and define the features of the thing that they're creating.
It takes a little recoding to turn
static r or python code into a shiny application but it works well with tidy and clean data that means the opposite is not true if you have massive raw data that is not pre-processed and analyzed building a shiny app is not a good idea the output is html and is created basically using shiny syntax you don't need to know any html or css to create it although if you know you can enrich and
make the app look nicer, but you don't need to know any of those.
And can we run locally or on a server?
If one is running on a server, as I just explained to you, the users can interact with this webpage and the code runs on the background in the server.
So now I'm going to quickly show you this app and show you some of its features.
For a moment, I thought it's one of those moments that you face an error in the wrong time.
So this is the app.
It has five main tabs.
In the first tab, actually, students and users of the app can
get some background information as to what this is about.
And the data is organized according to date.
They can select a particular date and then load the data.
And then it shows them which high school has been involved, which area they have looked into, how many stations they have had, and how many observations they have recorded in those stations.
In the next tab, they can actually see a route of their trip.
And they can include timestamps and they can include the location of the stations where they have stopped and have done those measurements and what they have measured.
And at the same time, they can see the observations that they had and the things that they have recorded and uploaded on Doobnots for Hava website.
They can see them organized here.
And at the same time, they can see this visualization of the zoner signal, which in this case, this is depth and this is time.
And the color here is a proxy for the density of the object that the wave sound is being reflected from.
So the denser the object, the color is stronger in this case.
And yes, this is just for those students to look and this is some bio marine biologists are working on this to actually enrich it and make it more understandable for those students.
In the third tab, after loading the data, which hopefully takes a few seconds, they can see the
the parameters that have been extracted and they can visualize those parameters and once they press this play button the trip sort of assimilates their trip the route starts and then the parameters start changing this way they can sort of track how those parameters have been changing along their trip for example this is
fish concentration, plankton concentration, humidity, no, sorry, temperature, humidity, and pressure, and then the depth.
This is like a monitoring tool to figure out what has been happening throughout the trip.
In the fourth tab, they can perform some analysis and basically
look at the correlations of those parameters that have been extracted.
For example, if someone is interested to know how fish concentration and plankton concentration are correlated, they can easily see it like this.
They can fit a line.
And later, if they want to see how this is, for example, correlated to the depth,
It can be visualized like this.
There is a clear correlation between fish and plankton.
And the deeper the water, the more fish and plankton you find.
This is just an example of an exploratory data analysis.
And if they want to, they can actually create a static plot of this dynamic plot and then continue.
just changing the parameters.
For example, if you are curious to know where they could have found more fish, they can do this.
If they remove the line, and instead of half bone, they color and size it with concentration of fish to make it clear.
Here you can see that in some areas around here and here, they could have found more fish or here, and then they can actually match it with the trip route here and basically get a sense of fish concentration.
These are just examples that I'm coming up with.
I don't know how they're using it and how they're going to use it in
their educational material.
So I don't want to get too distracted with these here.
And there's a fifth tab that actually visualizes some data from this lander complex that I was talking about that I don't want to use more time to talk about this.
But we have created this temporary account.
In case you are curious and you want to interact with this application, please feel free to go to this address and use this username and password and interact with this app.
And please know that this is work in progress and it would be nice to receive any feedback.
If you have any comments, ideas or questions, please feel free to send me an email.
I will be happy to discuss further.
And it will be available until tomorrow at 2pm for 24 hours.
So this is my last slide.
There are four partners for this Insight Osofjord.
D-Science, IT department, Konzberg Discovery and Inspira Science Center.
Svenja Kristiansen and Knut Wogen have been contributing to this project massively.
And we have been getting lots of help and support from different sections in IT department.
which was around building up these servers and dealing with EduCloud.
And that's Andreas and Trond from operating system, Shur, Halvor and Frank from application platforms.
So I acknowledge and appreciate their support and thank you for your attention.
This is awesome, definitely.
So will there be any code back in Oslofjorden?
Yeah, that's... You don't know anything about the results?
No, I am actually very literate when it comes to the biology of it.
I don't know so much.
You are an IT man, because you have programmed this web application.
Yes, but we have a marine biologist currently actually working on the biological aspect, and she will figure out much more when it comes to the biology.
And this is about creating new researchers, maybe.
You were talking about kids from Vidagående Skole, and so that is like a Tamsunds Oppdrag or something like that you are working on at Matnat, because this is a Matnat project.
Yes, if you consider design as a part of MATLAB.
Sorry for asking.
These are probably a bit over my head.
So when Vidagående skoleelever is going on excursion, they are collecting all this data.
They deliver it via the APIs to EduCloud.
And you are in a database or as piles?
Yes, they don't deliver anything.
They actually, those students, together with the biologists on board, they register parts of this data on a website called Dugna Turhavet.
The rest of data goes through different sources, including an Azure server.
And those students don't deal with that.
That's the Superior Science Center and Konzberg Discovery.
And then we use APIs to pull those data from different sources into the cloud.
And the data that the students create, those observations, that's part of it.
And that's a separate API.
And we collect all together and combine it.
And then make it available via this web application that we already have.
And many people have been logged in now, I can see.
But you also talked something about that the ferries in the Oslo fjord is collecting data.
Danske Borten, Kjell Feria.
To be honest, I don't know which boats they are.
Probably some of those that actually go through some regular route.
They are equipped with those sensors and they collect the data.
The data is coming.
We haven't had the chance to actually explore it yet, but that's the future plan.
That's cool.
So any questions?
No, I think I would just say thank you.
Yes, thank you.