so thank you so much for inviting me as an advanced user of TSD so to introduce myself I currently
work as an associate professor at the center for bioinformatics and departmental pharmacy at the
University of Oslo but I have the last decade worked at cancer ratio Norway and still do as a
part-time position and here's where we started our journey with TSD so what I will talk about today is how
cancer registry of Norway went from small to big data and our needs and solution architecture
projects how we train and troubleshoot our users appnode that introduced and how we use our Computing resources
the cancer registry of Norway I will just introduce it to you for those who don't know what it is
it's the register see of all cancer incidence in Norway so that's how maybe many people know
it but cancer registry also runs the screening programs in Norway and it has active research department
so from all of these departments at the cancer registry of Norway we produce a lot of sensitive data it's
the health record for the cancer cases it's the screening information and the research
that we do and I put small in quotes here because it's high quality data that has been curated
several times over and it would even after decades of collecting
data it's still quite small it fits to us some drive if you like but this is very valuable data
but progress happens and also the cancer registry has started to produce big sensitive data
specifically because the cancer registry of Norway also has a large bio banks where we use those
biobanks to produce omics data
so genomics epigenomics transcriptomics microbiome data and so on so these are Big Data and this is
also the focus of my own research group so so when we started a large-scale omics and other big data
production we had a need to complete and analyze this data and store it so the cancer registry
set up their own server inside what's already quite high security data environment and that
worked really well so we needed another one because we produce more and more data and then we needed
another one and then my research group and other groups that has already grow even further and we
produce even more data so we want it even more service but then
I researched our IT department said well let's look at this our own servers they are secure they are
quite nice and fast we have root access as Reserve researchers which is really nice or at least a
few of us had and we built our computer server park exactly how we needed it so it was quite
flexible to our needs
but our IT Department said this is not scaling properly if you need more and more in this this
creates a huge demand for maintenance and it's not really economically viable either so no you don't
get more servers what we have to do is find another solution and for the cancer registry it was an easy
choice to move into TSD for our needs so we created
cancer registry of Norway or kreftregisteret which we short quote KRG at TSD our own project our needs were of
course a high-security we have sensitive data we need a lot of community computational power because
we had our own server park so we were not used to queues so that we don't want queues we need HBP HBC
environment for all makes analysis and machine learning we have a biomimetic experts in our groups
so we need flexibility and software containerized movement of pipelines and so on we need to mirror
a lot of repositories but we also have a lot of novices people who are starting up with
large-scale analysis so we need familiar environments and it should be easy to use
and we need to have an computer environment where experts and novices can meet and we can also meet
with other collaborators and we need to collect survey data because we also do that at cancer registry of Norway
so the solution was to buy dedicated resources within TSD so we need our own computer course so
we didn't have to be
in any cues and we wanted a appnode which is a powerful computer where we can run graphical
interfaces and also have no queue and for the machine learning and artificial intelligence people we
needed to be use for the familiar environments we wanted Linux and Windows login notes because then
people see what they usually
see at their own computer and for this we wanted to give this to a lot of the projects at the Cancer
registry Norway and each project has their own documentation and ethical approvals and so on
so usually each project at TSD is their own project but here we wanted to share all of these
resources that we need
between many different projects so that means that the cancer registry at okay we can put money
into this but it should be shared but then we take care of all of that approval
of new users and projects
so we set up a file structure and permission system inside our project that the cancer registry run
on top of the TSD portal for new users
and by combining several project into one we can also collaborate on how to setup software and
solutions technique
so this is how it looks for us inside our own project today we have 53 users that are active on
on our project almost half of them are linked to my research group so this is a large group that
collaborates but do have different projects
and we meet TSD in the same way many others do we have a login node or I actually have several we
have a couple of Windows login nodes and we have a Linux login nodes so that serves all
of our needs and then we have an app node which we can run graphical programs on we have a submit
node and we can send those you can send jobs
through SLURM which many other uses too as well and we have access to different disk systems here as
well so this is how it looks for us
since we have many projects we needed to organize our folder systems within our project so each
section that needed TSD has their own overall folder where all the users get access to that and
the section leader which is or as someone they delegated this to can do the export which only
usually the PI can do
so and there's only one person at the castle Reggie who can approve new users and project and that
is also just after that the project coordinator have checked all of the approvals and have check that
the users have signed a non-disclosure documentation that you have to do at the cancer registry and
that all the approvals are in order
then the users get access to the project group which falls within one of those sections
so this is how we organize it
and we have many different projects underneath the same p1068 product that the cancer registry
for instance we have epidemiological project where data collaborations are really important where
they collect data from several different sources and need to share it between users not all at the
cancer registry and they collect
data also through nettskjema which is in active use and an example of a project is where they look
at cancer in offshore workers so this is a typical epidemiological project and there are several
others similar to this
and then we are quite active machine learning and AI group and some of them are working to
personalize screening so that means that they want to find exactly the best way of detecting cancer
at the earliest stage
they need high performance computing and GPUs for the AI analysis and for this the aim is for to
produce position for one of the project is to reduce position cervical cancer screening
and then there's the integrative genomics group which is my research group this research group just
does several different things we and I will give you some example of what we do on TSD shortly but
common for all of them is heavy computing so we need high performance computing we need to
collaborate between several different
users so meet me in the team work together on the same document on the same analysis we need the
graphical interface a lot of machine learning and the aim is very often biomarker discovery for
early cancer detection and we do a lot of omics so genomics epigenomics and so on
so I will give you an example of one of our projects that will run on TSD
so one of the larger project or I think this is the largest project is called CRC biome here we have
12 users working tightly together they have different backgrounds some are bioinformatics
some background from nutrition clinical background they are researchers postdocs data
managers PhDs and we have several master students in this project
so CRC biome our primary objective is identification of advanced colorectal lesions so this is to
correct the cancer based on the screening individuals get microbiome demographics and lifestyle so
get microbiome that's the bacteria and viruses living in your gut and we think that can be used to
predict colorectal cancer at an early stage
so why do we need TSD for this
okay so let's look at our study design so this study it's based on the BCSN
this is the randomized screening trial that's ongoing right now and this involves a lot of
individuals 150 000 were randomized either to sigmoidoscopy where they look through the first
part of the colon for cancer or they were given faecal
immunochemical home-test where they look for occult blood in their stool to the participants and what
we did is we invited those who are positive on this test for further data collection so we sent them
a lot of survey questionnaires and we didn't use the nettskjema for this
because this was somewhat older population and we didn't know about it and next time we will use
nettskjema for this since all these were positive on this blood in stool test they were sent to
colonoscopy so at the hospital they look through the entire colon but this there were tissue samples
and we ask for additional tests of this gut plus the big test
and we will be passive follow-up of this so we will collect data from the cancer registry of Norway and
the prescription history of all the participants in our studies this is now completed in inviting
people to this study so we have 1640 individuals so there's quite a lot of people invited we have
large survey data so there's lots of questionnaire data in
database they have clinical data which is highly sensitive we have registry data both from cancer
and prescription drugs and we take the the gut samples that fit samples and we do a shotgun
metagenomics from some of our participants that means we sequence the bugs and they've done that for
2099 samples so that's quite a lot
and we process this with in house and published pipelines on TSD
so shotgun metagenomics I can just briefly introduce you to this so so that means that we take all of
the bacteria and viruses that are in the gut they have DNA and we extract that we sequence that in
short sequences then we on TSD assemble them into full genomes again so piece them together
and then do data analysis so all of these processes are quite computer intensive
so when we've done all of this the piece the DNA together and figure out which bacteria each piece
belong to we try to estimate or classify the colonoscopy result which is telling us if they have
cancer or early stages of cancer based on gut microbiome
so we used to get microbiome as an input for machine learning analysis and for this we have established
our machine learning pipeline on TSD so this is just a flow chart of our pipeline and this is
something which is ongoing work we are now currently training our machine learning models 70% of our
data set and will later on do 30%
test this on 30% of the data set which we call the test set
so this is what we are doing but we also want to look into viruses in our gut microbiome viruses can
be also predictor of cancer development but can viruses are much less known than bacteria so this
means that we have less knowledge to identify viruses so we need a more complex
pipeline to do this and
we have set that up ourselves this is just all the steps in that process I won't go through them but it's
quite intensive a lot of new tools and using that we see that we can identify over 600 different
viruses in the guts of our participants which is a very large data source for us
I'm not going to show you a lot of study results from this study because they are not ready yet but
teaser here our result is that this is the ranked which variables are most important in our data set
and on top we have the colonoscopy result which is actually what we want to predict that means that
they're likely some signal in our
gut microbiome that can be used to predict cancer at the earliest age
which is very promising for us
but this project is advancing even further and the research funding to also do other omics so we
will do epigenomics and transcriptomics and peace all of them together to get a better understanding
of how colorectal cancer develops so this is a large project which involves a lot of different
skills and a lot of heavy computing
so that means we also include a lot of Masters students into our project so that means
that we all the time have new users and we run into new challenges often so like I said we have
different backgrounds in our group we are bioinformaticians epidemiologist and master students and
our master students come from informatics from biology from pharmacy so they have different
backgrounds
to not overwhelm the TSD help desk with all of these new questions since we have so many users we
have our own backup system where we can teach our own users and help them when they run into problems
so we have a Slack group with more than 30 members on this Slack group where we can send messages when
things are not running as it should be
when we have problems when new users can't figure out how to do things the Slack group is very
good too and if someone sees a problem which we think is TSD but might be just our own issue
we can check if with other users if there's really an issue or have anyone else a solution to this
this is an active Slack group between a lot of the users at the Cancer registry also we saw so we
really appreciate all the help we get on the UiO web pages for TSD but there are project specific
problems that we want to document and solve and also we want to get our new users up and running
as fast as possible understanding the architecture that we have and the solutions that we have on
our project so we have our own TSD user manual which is a very much alive document you can see it's
just updated all the time this document has everything you need to get started using TSD so it
tells people where to find their files and software how to do data management
how to just log into our own notes it has background information on basic use and not so basic use
how to use Conda singularities Snakemake Nextflow and access on Appnode and so on but also the
more advanced use we have some on documentation on how to get your R version
that you want to do the machine learning applications and digital stations and so on so all of this
is here and all of the users contribute and ask questions in the document I think that takes away a
lot of the emails to the TSD help desk
and the very special thing that we have access to which is really what makes TSD super
user-friendly for us is our Appnode which I highly recommend but this is something that we have to
buy ourselves and the TSD is running for us
in this Appnode we can use all kinds of software with graphical interface the most useful software
here is R studio we have a lot of R users in our group so R studio is definitely something
that is highly used and running R studio rather than just R is really something that makes it
more interactive for us we can see and
evaluate our analysis along the way seeing the graphical seeing our plots and so on immediately
but we also run IGV and Cytoscape and several others this is a quite powerful computer with the lots
of CPUs and lots of RAM and here you don't need to use slurm so and we have so much computer power
that this has never liked overloading this has never been an issue for us but we do check if someone
is running something heavy we do check with the other users if that's okay
we have a how-to of course in our manual on how to log into our Appnode this is just by SSH and with
the white flag and so it's really simple to use once you get started
and this is how it looks like it looks like your normal R studio which is really nice where we see
our plots immediately and this makes the threshold of starting to use TSD much lower because this
looks like something you can do on your own computer
so but since we have a lot of different users and they all prefer different versions of R and R studio
and different packages we had to really go through a few iterations to make this work for us but now
it works really nice we can load R studio with the versions that we need and also with the
packages that we need and we have
waste to choose your own versions and also load for instance bioconductor and so on
another thing that we found demanding to start with is our large workflows so I showed you all of our
steps in our workflows we handle many many input files
we have quite complex workflows with lots of dependencies and each had different software
requirements and some of these many of these are very resource demanding jobs for us the solution is
Snakemake we have several users who are really skilled Snakemake users
and here we automate our jobs using also slurm and we can submit this to our cluster which is really
nice and we can make adjustment to conda environment and so on so this is something we do a lot
here again is an example of a Snakemake workflow Snakemake works based on rules
so you have some input files you have some output files and the jobs you want to do and then the
output is input of the next step and so on so we specify what resources we want for each rule we
have threads memory time etc
and which software packages to use and we use a lot of from conda so you can set up the your rules
locally or using cluster so this is what we do a lot
so we here is an example where you can see how we set the number of cluster jobs used parameters and
condo environment we all put this in our command line and run our jobs
one of the issues that we met was that if we put all of our pipeline the Snakemake into
one slurm job it was running longer than a job could and it timed out for us so now when Snakemake
is sending multiple jobs it becomes more manageable but also becomes very complex
this is an example of
very complex pipeline that we use its Atlas pipeline which is published here and this we run with slurm
and each step is a slurm job we have more than 2,000 samples with lots of lots of information each and
when we send all of these steps to slurm it creates many many many jobs
but this works nicely for us all in all it takes multiple processing and here TSD has been vital to
be able to do this
so this is just a nice picture of the research group collaborative and some friends among them
we see the super users in a project that has been vital to get up things up and running as we would
like it inside TSD specifically Einar Birkeland here Sinan Umu here
Marcin over here has been the three postdocs now researchers in the group whose
solved a lot of our issues and now make TSD exactly what we need but we have other also who has
contributed and specifically Jan Nygård which is the head of IT at the cancer registry has been really
vital to setting this up so this I would like to say thank you for your time and I will answer
questions if there are any
thank you very much Trine
you have it's very impressive what you have done how you use all these more complex ways in TSD
I know you have a lot of data, actually terabytes of data
so its a lot of data the cancer registry of Norway it is not
inside these this is just a project connected to right yeah so the cancer registry
Norway still runs a lot of its own servers and that is the main thing that the cancer registry
does like collect all the cancer information in Norway and so on all these screening program
everything is run by the cancer registry of norway themself they has a large IT department who has
set that up so but it's really the project the research project that has moved to TSD
you said a lot about R studio and there will be more about the
R studio and are marked down how we use that on advanced projects and Thursday actually so let's see
people are now often questions and not only complaining that their stream didn't work
anymore we have someone giving
feedback saying great presentation Trine so I guess that's kind of an applause so there was one
question about all the CR biome participants is 1640 participants how you administer all these
participants do you have like a database for this entire TSD or how do you do
we have some old-fashioned things we have consent on paper and the questionnaires on paper still but
then we have scanned it and imported it to TSD and it's it's not databases we actually use R studio
for almost anything so it's just R studio files and linking data there
okay so it's more like a combination of extremely high and complex technology and very basic
research technology as some paper so I hope you can try to use more digital content for example
and stuff like that after a while
how many users do you have in scientific 56 for that when their number of local users in scientific
53 I think and
he said how does that work with so many people inside the same project there has to be any issues there
so we learn as we go so in the beginning it's just but we were used to working on
the same server inside the the cancer registry so we moved the way of working by communicating
between ourselves to just TSD so that culture in a way was already set there's some issues with so
many users on one project we are we are not
we have enough Computer Resources but actually the login notes are very highly used and sometimes
too many users on the login notes even though no process that are running there that's
something we need to scale in the future it's so many people on the same login notes but it works
really nice I think and we learn from each other so that's also an advantage you are not
on your own solving new things that you want to do inside TSD
and then there also was a small question maybe Leon can offer as well because he is having an exhibition about the
cost of buying these kind of services from us what is the cost of all this Appnode for example
yeah so I think the cost is also related to how much computer resources you need I
think Leon and Gard can probably answer
that but we had to make a decision one and a half year or so two years ago
probably two years ago already do we put the money in just buying new servers or send the money to
you and you set up what we need and the customizable you were really helpful
for us for that TSD really just set up what we needed the cost I think has changed so
our costs are outdated but really just giving the number what type of money you have and what you need
and see if that fits together that was the how we did it and we are able to
skip now can we can certainly give you a 60 terabyte no problem
Leon will tell a lot more about this in his next destination how
we now scale so I think we just say thank you very much a great presentation