-1

I've been reading about machine learning models deployment and pipelines and so far most of the sources suggest that the data should be ingested from some sort of cloud based storage or source, be it AWS S3, Kaggle, Bigquery or whatever else.

Now the thing is that in my company we analyze sensitive client data which i think should not be stored in the cloud as its a potential security threat, or at least it should not leave the country/EU because of GDPR.

So given this how machine learning pipelies can use offline, local data to work?

Improve this question
1

Four options:

  • you have plenty of means (e.g you’re a government, a bank, a fortune 500): you build your own private cloud. You could even tender and let a cloud provider do it for your in your own facilities under your control and clearances.
  • depending on how or where the processing is done, you could encrypt the data with a key that does not leave your facility. The data in the cloud would then be unusable for anybody else (except perhaps the NSA).
  • depending on sensitivity, you could anonymize the data. This makes it usable only if you know how to desanonymize it (which is usually based on data staying in your premises). There are a couple if tricky risks to cope with (e.g. digital twins that could help to identify someone base on a combination of properties).
  • you accept the risk, find the right lawyers to be sure the cloud provider complies with all legal requirements and security standards and can be sued in case of breach. In this case, you could introduce some dummies in your data on monitored email addresses, so that you can spot data theft (e.g. sundden spam campaigns on the dummies)
Improve this answer
0

There is no such thing as “the cloud,” it’s just somebody else’s computer

(This is a common saying, the version above came from an article by Hayato Huseman)

"The cloud" is just a fancy way of saying that the data is stored on someone else's server, somewhere. Often, you don't know where the server is, and it could be moved without telling you.

If you don't want to do that, create a "private cloud", which is just a fancy way of saying that you are storing the data on your own server, in your own server room.

Improve this answer
5
  • So you mean I should somehow figure out how to inject data locally? Or you mean I should setup local on premises server that where I should store the data and make it available to others I want?
    – Alex T
    Dec 4 '19 at 11:42
  • @AlexT You could buy "cloud" software to run on your own server, so that it looks just like any other cloud service to your application software. Or just ignore the advice to use "the cloud" and just store the data on your own disks. People like to make distributed computing sound like magic in order to sell services to people. There's nothing to stop you doing it yourself and saving the money. You just have to be prepared to work out for yourself howto do it.
    – Simon B
    Dec 4 '19 at 21:39
  • the thing is this is what we do and been doing(storing data on our own disc) except, that now we create data management pipelines that should be reproducible across different machines-hence the need to store the data somewhere that could be accessed by other machine
    – Alex T
    Dec 5 '19 at 9:15
  • @AlexT Do you have any problem (apart from "buzzword compliance") that can't be solved using a server with a shared network drive, or a NAS box? And "server" can be just a spare PC with a big hard disk.
    – Simon B
    Dec 5 '19 at 9:55
  • I'd say its more like Im not sure how to approach this local server setup since all I did before was use ready to go services like Azure Storage or AWS S3.
    – Alex T
    Dec 5 '19 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.