Return to Question

Looking for the correct architecture -

I have a serverless app, hosted at AWS. It uses Lambdas and several DynamoDB tables for most of the BE logic (managed with aws-amplify).

I want to add a feature where users can upload CSVs, see them as a table on the app, and create a simple public API to fetch one row, based on ID (no need for more complex queries). Structure of the CSV (columns) varies with each upload.

Each users will add about 0-10 CSVs, each CSV will contain 3-20 columns and around 1k-100k rows.

How should I build the database part? (not limited to Lambdas/DynamoDB)

Thanks

EDIT

The solutions I had in mind are:

1. Create a new table (sql/document) for each CSV upload, and save the name of the table under user.csvs[].

This way I'll have huge amount of tables. Is that a reasonable solution?

2. Add all CSV data to a document db, e.g. --

user { 
name: "john",
csvs: {
  csv123: {
    id345: {col1: 'x', col2: 'y'},
    id678: {...},
    ...
  }
 }
}

What should I index in this solution for best performance?

Looking for the correct architecture, that should take into consideration - future edge-cases, bugs and pitfalls, performance, cloud pricing, ease of building and maintaining and security.

I have a serverless app, hosted at AWS. It uses Lambdas and several DynamoDB tables for most of the BE logic (managed with aws-amplify).

I want to add a feature where users can upload CSVs, see them as a table on the app, and create a simple public API to fetch one row, based on ID (no need for more complex queries). Structure of the CSV (columns) varies with each upload.

Each users will add about 0-10 CSVs, each CSV will contain 3-20 columns and around 1k-100k rows. Adding CSVs takes place once a month/week, reading a line with the API happens 10k-100k a day.

How should I build it? (not limited to Lambdas/DynamoDB)

The solutions I had in mind are:

1. Create a new table (sql/document) for each CSV upload, and save the name of the table under user.csvs[].

This way I'll have huge amount of tables. Is that a reasonable solution?

2. Add all CSV data to a document db, e.g. --

user { 
name: "john",
csvs: {
  csv123: {
    id345: {col1: 'x', col2: 'y'},
    id678: {...},
    ...
  }
 }
}

What should I index in this solution for best performance?

3. upload the file to a bucket, and create a lambda that opens it with every request, and returns the requested line. (this option skips DB indexing)

Hope to learn from your experience.

Looking for the correct architecture -

I have a serverless app, hosted at AWS. It uses Lambdas and several DynamoDB tables for most of the BE logic (managed with aws-amplify).

I want to add a feature where users can upload CSVs, see them as a table on the app, and create a simple public API to fetch one row, based on ID (no need for more complex queries). Structure of the CSV (columns) varies with each upload.

Each users will add about 0-10 CSVs, each CSV will contain 3-20 columns and around 1k-100k rows.

How should I build the database part? (not limited to Lambdas/DynamoDB)

Thanks

EDIT

The solutions I had in mind is:

1. Create a new table (sql/document) for each CSV upload, and save the name of the table under user.csvs[].

This way I'll have huge amount of tables. Is that a reasonable solution?

2. Add all CSV data to a document db, e.g. --

user { 
name: "john",
csvs: {
  csv123: {
    id345: {col1: 'x', col2: 'y'},
    id678: {...},
    ...
  }
 }
}

What should I index in this solution for best performance?

Looking for the correct architecture -

I have a serverless app, hosted at AWS. It uses Lambdas and several DynamoDB tables for most of the BE logic (managed with aws-amplify).

I want to add a feature where users can upload CSVs, see them as a table on the app, and create a simple public API to fetch one row, based on ID (no need for more complex queries). Structure of the CSV (columns) varies with each upload.

Each users will add about 0-10 CSVs, each CSV will contain 3-20 columns and around 1k-100k rows.

How should I build the database part? (not limited to Lambdas/DynamoDB)

Thanks

EDIT

The solutions I had in mind are:

1. Create a new table (sql/document) for each CSV upload, and save the name of the table under user.csvs[].

This way I'll have huge amount of tables. Is that a reasonable solution?

2. Add all CSV data to a document db, e.g. --

user { 
name: "john",
csvs: {
  csv123: {
    id345: {col1: 'x', col2: 'y'},
    id678: {...},
    ...
  }
 }
}

What should I index in this solution for best performance?

Looking for the correct architecture -

I have a serverless app, hosted at AWS. It uses Lambdas and several DynamoDB tables for most of the BE logic (managed with aws-amplify).

I want to add a feature where users can upload CSVs, see them as a table on the app, and create a simple public API to fetch one row, based on ID (no need for more complex queries). Structure of the CSV (columns) varies with each upload.

Each users will add about 0-10 CSVs, each CSV will contain 3-20 columns and around 1k-100k rows.

How should I build the database part? (not limited to Lambdas/DynamoDB)

Thanks

EDIT

The solutions I had in mind is:

1. Create a new table (sql/document) for each CSV upload, and save the name of the table under user.csvs[].

This way I'll have huge amount of tables. Is that a reasonable solution?

2. Add all CSV data to a document db, e.g. --

user { 
name: "john",
csvs: {
  csv123: {
    id345: {col1: 'x', col2: 'y'}
  }
 }
}

What should I index in this solution for best performance?

Looking for the correct architecture -

I have a serverless app, hosted at AWS. It uses Lambdas and several DynamoDB tables for most of the BE logic (managed with aws-amplify).

I want to add a feature where users can upload CSVs, see them as a table on the app, and create a simple public API to fetch one row, based on ID (no need for more complex queries). Structure of the CSV (columns) varies with each upload.

Each users will add about 0-10 CSVs, each CSV will contain 3-20 columns and around 1k-100k rows.

How should I build the database part? (not limited to Lambdas/DynamoDB)

Thanks

EDIT

The solutions I had in mind is:

1. Create a new table (sql/document) for each CSV upload, and save the name of the table under user.csvs[].

This way I'll have huge amount of tables. Is that a reasonable solution?

2. Add all CSV data to a document db, e.g. --

user { 
name: "john",
csvs: {
  csv123: {
    id345: {col1: 'x', col2: 'y'},
    id678: {...},
    ...
  }
 }
}

What should I index in this solution for best performance?