How do securely store user files with node, s3 and Stormpath?

How do securely store user files with node, s3 and Stormpath?

  • 2016-09-22
  • 650

There are a lot of redundant problems you need to solve as a web developer. Dealing with users is a common problem: storing them, authenticating them, and properly securing their data. This particular problem is what we here at Stormpath try to solve in a reusable way so that you don’t have to.

Another common problem web developers face is file storage. How do you securely store user files? Things like avatar images, PDF document receipts, stuff like that. When you’re building a web application, you have a lot of choices:

  1. Store user files in your database in a text column, or something similar
  2. Store user files directly on your web server
  3. Store user files in a file storage service like Amazon S3

Out of the above choices, I always encourage people to go with #3.

Storing files in a database directly is not very performant. Databases are not optimized for storing large blobs of content. Both retrieving and storing files from a database server is incredibly slow and will tax all other database queries.

Storing files locally on your web server is also not normally a good idea. A given web server only has so much disk space, which means you now have to deal with the very real possibility of running out of disk space. Furthermore, ensuring your user files are properly backed up and easily, consistently accessible can be a difficult task, even for experienced engineers.

Unlike the other two options, storing files in a file storage service like S3 is a great option: it’s cheap, your files are replicated and backed up transparently, and you’re also able to quickly retrieve and store files there without taxing your web servers or database servers. It even provides a fine-grained amount of control over who can access what files, which allows you to build complex authorization rules for your files if necessary.

This is why I’m excited to announce a new project I’ve been working on here at Stormpath that I hope you’ll find useful: express-stormpath-s3.

This is a new Express.js middleware library you can easily use with your existing express-stormpath web applications. It natively supports storing user files in Amazon S3, and provides several convenience methods for directly working with files in an abstract way.

Instead of rambling on about it, let’s take a look at a simple web application:

'use strict';
 
const express = require('express');
const stormpath = require('express-stormpath');
const stormpathS3 = require('express-stormpath-s3');
 
let app = express();
 
// Middleware here
app.use(stormpath.init(app, {
  client: {
    apiKey: {
      id: 'xxx',
      secret: 'xxx'
    }
  },
  application: {
    href: 'xxx'
  }
}));
app.use(stormpath.getUser);
app.use(stormpathS3({
  awsAccessKeyId: 'xxx',
  awsSecretAccessKey: 'xxx',
  awsBucket: 'xxx',
}));
 
// Routes here
 
app.listen(process.env.PORT || 3000);

This is a bare-bones web application that uses Express.js, express-stormpath, and express-stormpath-s3 to provide file storage support using Amazon S3 transparently.

This example initialization code requires you to define several variables which are all hard-coded above. This minimal application requires you to:

  • Have a Stormpath account and have already created a Stormpath Application
  • Have an Amazon Web Services account, and to have created an S3 bucket in the US Standard region (this is where your user files will be stored)

Assuming you’ve got both of the above things, you can immediately start using this library to do some cool stuff.

Uploading User Files

First, let’s take a look at how you can store files for each of your users:

app.get('/', stormpath.loginRequired, (req, res, next) => {
  req.user.uploadFile('./some-file.txt', err => {
    if (err) return next(err);
 
    req.user.getCustomData((err, data) => {
      if (err) return next(err);
 
      res.send('file uploaded as ' + data.s3['package.json'].href);
    });
  });
});

This library automatically adds a new method to all of your user Account objects: uploadFile. This method allows you to upload a file from disk to Amazon S3. By default, all files uploaded will be private so that they are not publicly accessible to anyone except you (the AWS account holder).

If you’d like to make your uploaded files publicly available or set them with a different permission scope, you can easily do so by passing an optional acl parameter like so:

app.get('/upload', stormpath.loginRequired, (req, res, next) => {
  // Note the 'public-read' ACL permission.
  req.user.uploadFile('./some-file.txt', 'public-read', err => {
    if (err) return next(err);
 
    req.user.getCustomData((err, data) => {
      if (err) return next(err);
 
      res.send('file uploaded as ' + data.s3['package.json'].href);
    });
  });
});

The way this all works is that all user files will be stored in your specified S3 bucket, in a sub-folder based on the user’s ID.

Let’s say you have a Stormpath user who’s ID is xxx, and you then upload a file for this user called some-file.txt. This means that your S3 bucket would now have a new file that looks like this: /xxx/some-file.txt. All files are namespaced inside of a user-specific folder to make parsing these values simple.

Once the file has been uploaded to S3, the user’s Custom Data store is then updated to contain a JSON object that looks like this:

{
  "s3": {
    "some-file.txt": {
      "href": "https://s3.amazonaws.com/<bucketname>/<accountid>/some-file.txt",
      "lastModified": "2016-09-19T17:59:22.364Z"
    }
  }
}

This way, you can easily see what files your user has uploaded within Stormpath, and link out to files when necessary.

The express-stormpath-s3 documentation talks more about uploading files here.

Downloading User Files

As you saw in the last section, uploading user files to Amazon S3 is a simple process. Likewise — downloading files from S3 to your local disk is also easy. Here’s an example which shows how you can easily download previously uploaded S3 files:

app.get('/download', stormpath.loginRequired, (req, res, next) => {
  req.user.downloadFile('some-file.txt', '/tmp/some-file.txt', err => {
    if (err) return next(err);
    res.send('file downloaded!');
  });
});

As you can see in the example above, you only need to specify the filename, no path information is required to download a file. This makes working with files less painful as you don’t need to traverse directory paths.

You can read more about download files in the documentation here.

Deleting User Files

To delete a previously uploaded user file, you can use the deleteFile method:

app.get('/delete', stormpath.loginRequired, (req, res, next) => {
  req.user.deleteFile('some-file.txt', err => {
    if (err) return next(err);
    res.send('file deleted!');
  });
});

You can read more about this in the documentation here.

Syncing Files

Finally, this library provides a nice way to ensure your S3 bucket is kept in sync with your Stormpath Accounts.

Let’s say you have a large web application where you have users uploading files from many different services into S3. This might result in edge cases where files that were NOT uploaded via this library are not ‘viewable’ because the file metadata has not been persisted in the Stormpath Account.

To remedy this issue, you can call the syncFiles method before performing any mission critical tasks:

app.get('/sync', stormpath.loginRequired, (req, res, next) => {
  req.user.syncFiles(err => {
    if (err) return next(err);
    res.send('files synced!');
  });
});
 

This makes building large scale service oriented applications a lot simpler.

You can read more about the sync file support here.

Wrapping Up

Right now this library is available only for Express.js developers. If you find it useful, please leave a comment below and go star it on Github! If we get enough usage from it, I’ll happily support it for the other Stormpath web frameworks as well.

Suggest

Learn Nodejs by Building 12 Projects

The Complete Node JS Developer Course

Learn and Understand NodeJS

Build an Amazon clone: Nodejs + MongoDB + Stripe Payment

Angular 2 and NodeJS - The Practical Guide to MEAN Stack 2.0