For the second major version of TYPO3 Flow we integrated Composer, the upcoming de-facto standard for package management in PHP. I've become a big fan of its dependency management capabilities and especially the ability to pull almost any package from packagist.org into my Flow applications. But enterprise projects need a little more than what comes with the starter kit.
Needs of Big Projects
While the average Joe Developer (me included) loves the ability to get started with a simple curl -sS https://getcomposer.org/installer | php and install all necessary dependencies of an application by executing composer update, this frequently causes nightmares in operations of bigger companies. In some environments and with certain security standards, you just want to have as much control as possible over what ends up on your production servers.
What we end up doing in these projects is to manually create a local fork of every single package involved and pull in new changes in defined iterations. For the application's composer manifest, we disable packagist lookups and only use our own Satis-generated repository.
But it is tedious to do this manually and especially in projects with dozens of developers and more than 50 packages involved, it can be hard to keep track of all dependencies and if they (still) make sense.
Composer Enterprise UI
With more and more big companies using composer, this just has to become a common requirement. Wouldn't it be nice to have a "Composer Enterprise UI" which you can install locally and let it display all packages and licenses you currently use in your projects? See at one glance which forked repositories are clean and which ones contain hotfixes?
Given at how much time IT departments spend on setting up and maintaining a custom structure, this should definitely be something a developer or shop could earn some money aside.
Talking about Composer over a beer with Jordi Boggiano
Signed Composer Packages
One topic which has been discussed in the past already is the demand for signed composer packages. From my experience what disturbs operations most (and what they get as their first impression about Composer) is the one-liner to let Composer install itself. What they ask for (and I think, rightly so) is a way to at least be sure that a given package is exactly in the state a certain developer released it. For this target group, pre-packaged (RPM, dpkg, ...), signed versions of Composer and signed composer packages would be a big relief and enabler.
How do you deploy Composer in enterprise projects? Which other ways can you imagine for getting more control over what gets installed?
Rens
Hey Robert,
Another way of being sure what ends up on the production server is not using composer there at all. To me even having a git clone on the production server is something to discuss. I know about companies who will not allow git clones, but prefer using rsync to deploy.
It's my personal fear that a lot of tools used by developers (like git, composer) end up on production machines because of 'convenience'. I will probably do the same as it takes less time. But is this really the way to go?
Greetz
Robert Lemke
Hey Rens,
I think it depends on who is executing git: if you let a tool like Capistrano or TYPO3 Surf control the deployment, I don't see a risk in relying on git. It's important though that the deployment to production is the exact same like the one you committed to your staging environment.
Rsync wouldn't do it for me, as I need a more capable tool for preparing a new release on the target environment and then switch a symlink when its ready.
Needless to say that I'm a fan of production servers where no humans have a shell account for ;-)
Václav Novotný
"...install all necessary dependencies of an application by executing composer update..."
Just for sure, do you use "composer update" or "composer install" for installing of packages? Using "composer update" for installing of dependencies is very unpredictable but if you will use "composer install", all versions are hardcoded in composer.lock file.
More about that is in this great presentation - http://www.slideshare.net/rdohms/composer-putting-dependencies-on-the-score
Robert Lemke
@Václav yes, I wrote "composer update" on purpose because that's what most developers do during development. The challenge for operations is that developers pull in new versions of external dependencies and write them into composer.lock which then is used for deploying the application.
I guess you can trust most developers to pull in only versions which make sense, but then again I can comprehend the wish by operations to have a say in that, too.
And thanks for the pointer to the slides! Rafael's presentation was a nice one and certainly recommended to everyone new to Composer!
Václav Novotný
@Robert: I see. So you are right, that can be annoying for operations.
Rens
Just to point out: I would use rsync for transport only, but not to my document root of course ;) I would definitely use TYPO3 Surf. I'm not sure if the rsync support in Surf is ready yet, but will definitely have another look at that soon.
Regarding the composer update. It's so easy to just to 'composer --dev update' when just one package has to be updated. So yes, you should be able to trust the developer, but can you really? At least I plead guilty for having pushed lockfiles with updates I just took for granted... Maybe the interaction during an update should be extended a bit, like actually asking for an update to be applied if found.
Lukas
I agree such a tool would be useful. I guess http://desktop.sensiolabs.org is going in that direction.
With Enterprise deployment we usually associate a project independent sysadmin team, that is unwilling to do anything but follow very clear instructions (ie. they must be fool proof) and that really do not like accessing any data outside if their intranet and that are unwilling to change any of their internal processes.
For this setups we have often created custom tar package formats and installer scripts or ideally used .deb/.rpm packages. For the later there is a nice tool called fpm (https://github.com/jordansissel/fpm). See also:
http://blog.liip.ch/archive/2011/11/22/packaging-solution-for-php-projects.html
BTW you might also be interested in https://github.com/ekino/php-mirroring
Robert Lemke
Lukas, thanks for the pointers!
Sebs
I dont think just forking is not enough. Alone thats just snakeoil. If you fork a repo, then you have to codereview all stuff completely in order to be sure that its a "clean" state you are using.
A GUI is maybe a sales job helper, but I rarely saw enterprise environments to be maintained with that kind of tools. Maybe I am a OS afficiando where these tools are not that wide spread, but my perception is: The more Enterprise it gets, the less UI I find. However this might be my own experience.
In the end, using OS you build on a lot of instability and it is a good thing, not a bad one. But that, even in 2013, must reach the heads of the people steering the IT in these enterprises.
mablae
Hi!
I think the worse problem are untagged dependencies. If you have only dev-master this is really annoying.
If any packagist package would have proper versioning the problem of unreliable "composer update" would be gone, right? Just put proper version constraints to composer.json.
Another thing: Why should a devop use "composer update"? There is "composer install" to install exactly the packages locked in composer.lock.
Idea: A devops version of composer that doesn't do :update at all. So devops cannot destroy projects accidently.
Maybe other options related to devops could added, too.
What do you think?
mablae
david
One additional good reason to maintain local forks of packages is also that things can disapear on github. If a dev needs to install an old project for an emergency, he has no time to update to such changes on the fly.
The risk with local copies of course is to forget to track what upstream does... Tools that help syncing would surely make sense.
Ken
I use this tool https://github.com/researchgate/broker
Basically it creates a packagist like mirror of what's in your project's composer. I'm not sure if it can use composer.lock which is much better for mirroring the exact versions of packages.
Till
Maybe you guys want to add your two cents here:
https://github.com/composer/composer/pull/1567
This PR addresses the first part — archiving releases on your own infrastructure to be independent of sites like Github, Bitbucket, etc..
IMO mirroring your repositories is especially trivial with tools like Git, HG and probably not the scope of Composer. This doesn't address code audits, etc. — but that's another concern.
I also agree that signing packages should be something that is worked on next and I'd appreciate it if someone would share ideas they have to achieve this. It's another feature request that has been sitting around for a long time:
https://github.com/composer/composer/issues/38
Hrishikesh Lele
Thanks for this info as we have some of these Enterprise issues. - Hrishi