0

I'm currently writing an e-learing web-application. I'm sorry if this is information overkill but I think it's better to describe it in detail so you get the idea. I have made thought on every point and would just like to know if my approach is ok or totally off the mark for this is the first time I handle that much data "on the fly".

The e-learning courses contain files of different types, e.g. Word files, PDFs, Flash (yes, sadly no way around it), videos etc. and have 1 or more sets of tests, each containing questions where images can be attached (one each question) and answers of course (single- or multiple-choice), which also can hold images (one each) in addition or instead of pure text. Upon completion a PDF (certificate) is created and stored on the server for the user to download at any time.

Planned workflow:

  1. Course creation is handled similar to a checkout in online shops, meaning you click on "New course" and a window pops up and at the top you see a flowchart showing where in the creation process you are right now. The whole process should be dynamic so information is only stored in the final "checkout". Each part of the process is on a separate "page" but you can go back and edit things. It'll be visualized like this (I will add text descriptions to the points of course): Workflow
  2. First page contains only basic information, e.g. Name, version, short description and a list of topics being taught in this course.
  3. 2nd page is where the fun starts: Upload multiple files with description. These file uploads should not disable the use of the form. Videos shall be uploaded and encoded in background.
  4. 3rd page: Test-sets are created, this means categories (Test-sets) with subcategories (questions) and sub-subcategories (answers) also with multiple file uploads (images).
  5. 4th page: final overview and confirm.

How I want to handle things:

Data pre-storage

I would like to use temporary tables for this case to store the file paths and basic data. As the creation process can be cancelled at any time all files belonging to entries for this current creation process shall be deleted. This means creation starts --> Process is created in temporary table creation-processes. After the first page is filled the data is stored in a temporary table with fk to process id, same goes for any file uploaded.

Video encoding

As soon as a video file is uploaded I will run a background php-script which sets a database-entry in the temporary table for the video file and starts encoding via ffmpeg (3 files for each video file: mp4, ogg, webm). Uploaded files can have 4 states: "Uploading", "Processing", "Finished" and "Error".

Permanent storing

Upon completion the Data from the temporary tables is transferred into permanent tables and then deleted. Files still being processes (as video encoding can take some time) will also have their state transferred in the permanent table. The background-encoding-script checks for the filename (which is unique: timestamp+md5-hash of filename) in the temporary table first and then in the permanent table. If no entry exists in both it deletes the files it created.

Garbage collection

In addition to just using temporary tables I will have a permanent table "tmp_files" containing path information of ALL files being uploaded. The entries and the files listed in this table are deleted upon cancellation of the process, also after completion the entries are deleted (but not the files) for they are now stored in the regular table (course-files). So for things going wrong I can run a cron job every hour that compares the entries of "tmp_files" with the files in the regular table and the temporary table and if it holds an entry which is not present in one of the other tables this means there is a zombie file which must be deleted (together with the entry of course).

Well that's it so far. I don't know if questions regarding planning of software are welcome here but I really would like to know if this is a good way to approach this process or not and if I'm missing something essential.

  • 2
    The proper design of your software will become increasingly apparent as you complete your requirements gathering and begin actually building the software and solving problems. Offhand, I'd say that your thought pattern seems sensible, but a lot of it will change as you discover that some things won't work and others will need to be modified to fit your requirements. Despite your concerns about the amount of information in your question, this outline is still very general. – Robert Harvey Aug 27 '14 at 14:59
  • Why are you using temporary tables instead of status tables or a status field? – Robert Harvey Aug 27 '14 at 15:04
  • Well I'm actually in the process of building it. Listing of courses etc. is already implemented and I'm currently working on the creation form. But because of there is so much stuff going on in background I don't want to take the risk of putting so much time and effort into something that is plainly wrong by design so I thought I'd better ask for tips. :-P Thanks for confirming my think pattern though, knowing to not being totally off the mark is good. – Broco Aug 27 '14 at 15:04
  • Are you using any architectural patterns, like MVC? – Robert Harvey Aug 27 '14 at 15:04
  • @RobertHarvey I'm not using MVC but I guess I should start using it now. Until now there was no process that required that much information to be processed. I use temporary tables because I thought in case of a server crash they will be deleted from memory which means that if the server crashes during creation there will be no zombie files because if the garbage collector doesn't find entries for files neither in the temporary tables nor in the permant tables it will delete them. – Broco Aug 27 '14 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.