Once you start running a Flow application from multiple web servers you need to figure out where and how to store all the file uploads, documents and other assets. Setting up an NFS share as a common file base is possible but often cumbersome and, let's face it, so nineties. You really want to store all your persistent resources in the cloud and, as a bonus, use a content delivery network for high speed and reliable access by your users.
Store Content and Deliver Content
There are basically two scenarios we need to cover:
- instead of using the local file system (./Data/Persistent/Resources), we'd like to transparently store all "persistent resources" in a cloud location
- instead of publishing resources to the local file system (./Web/_Resources), we'll want to publish them to the cloud and use a URI provided by the CDN or cloud storage
These scenarios have been in the back of my mind from the beginning when I started designing the resource management in Flow. The plumbing is there, but it lacks the actual implementation and a few adjustments to really store and publish content to the cloud.
Make it Happen
These days I'm working on related parts in the Flow core. I will create a first prototype which can upload resources to Rackspace CloudFiles and use its publishing mechanism.
There's only so much time I can spend with it out of curiosity, so I won't be able to make this generic and polished enough for the common case. Unless … you need this for your own Flow or Neos project! Do you want to store resources in Amazon S3? Or use Akamai as a CDN? Get in touch with me if you need some cloud-resource functionality and have a budget for it.
Which cloud and multi-server features are on your wish list?
Olivier Dobberkau | 15.01.2013
Hi Robert.
Our TYPO3 CMS experience is pretty straight forward using Amazons Cloudfront. There is basicly no need for the CMS to be integrated with any fancy S3 Stores or so. There is an option called origin-server that requests the files from your Server and delivers them thru the CDN. In TYPO3 CMS just rewrite the assets to a special CNAME domain and then Cloudfront takes care of the rest. (We use this alot and use a regex extension called ja_replacer for it)
Where as I am unsure how to handle things like user upload (personalized assets) with Flow/Neos.
If you want some ideas about possible objectstores see:
http://wiki.typo3.org/T3A-budget-ideas-2013
Look for the FAL Adapter Ideas there.
And I am sure you could also join our CLOUDY project as Karsten did. :-)
http://forge.typo3.org/projects/show/extension-cloudy
Robert Lemke | 16.01.2013
Olivier, thanks for the pointer. However, this is mostly about handling persistent user or system generated assets, not so much about static assets like CSS files etc. Sure, we'll cover the latter as well, but the point is to store uploads in the cloud.
In addition, the resource framework in Flow is not really a file abstraction layer, but provides resources as objects. You can't move or rename resources in Flow. You just use an object which represents an image or a PDF and don't care where it is located.
From what I can tell, there is not much overlap with the FAL yet.
Martin Brüggemann | yeebase media GmbH | 16.01.2013
Great! We need this for one of our TYPO3 Flow based projects and will support you. I like the FAL and think its a big win for TYPO3 6.0 but it can't get it installed as a TYPO3 Flow package ;)
Olivier Dobberkau | 16.01.2013
Hi Robert. Ah! I must have missread some lines in your blog post...
If you would ask me which software to use then i would be saying: Riak is my first choice. Where as i must admit that i do know nothing about the plumbing us Flow atm and i really would enjoy to see more "birds-view" explanations (here and on typo3.org).
Jochen Rau | 16.01.2013
Olivier, Riak is an excellent choice for building a scalable, high-available storage. It has a sound architecture (implemented in Erlang) and a PHP client lib. More importantly--from a Flow perspective--it has a RESTful API. So, Flow could simply issue GET, PUT, POST, DELETE requests to handle its assets. This would decouple Flow from any concrete storage implementation.
Robin C.
Hey Robert, we are currently building a web application based on flow which needs a solution to store the persistent resources anywhere (elsewhere as the local file system). It's important for us to have a solution for this process. Can you refer some new ways for this?
Henk | 10.02.2014
Hi Robert,
Have you seen this? FAL Rackspace Cloud Files--> http://forge.typo3.org/projects/extension-fal_rackspace_cloud_files A co-worker of me is working on this.