how many instances of d are there in:
q: how can i reduce inode utilization on an ext4 filesystem? summary of my need
we put a large amount of files on a filesystem for analysis at a later time. we cant control how many files were going to have, and this one box needs access to all of them.
unchangeable limitations


*

*i cant change the inode limit. its ext4, and its the default 4 billionish

*there will always be a lot of files. the question isnt how to reduce the number of files; its how to circumvent the 4bn inode limit.

*i cant use network storage. this box lives in a data center and due to the staggering amount of existing data throughput, network storage is not an option.


my ideas


*

*i could mount a file as a loopback device in the location where were placing these files.


*

*pro: simple to implement

*con: another layer of complexity, but a pretty thin one.


*xfs. no inode limit.


*

*pro: this obviously just erases the problem.

*con: not sure how much flexibility ill have in making this change to a production system.



my question
what are some other stragies for circumventing this hard limitation? are there other benefits/drawbacks to the approaches ive mentioned? 

a: i would suggest that you use a network server with a filesystem designed to handle what you need. the first thing that comes to mind would be something that supports zfs (freenas and nexenta, though the free version of the latter has some limitations) or if you can afford it you can buy something like netapp.
i am less familiar with ufs available on freebsd etc., but heard that would work too.

a: info were missing... 


*

*distribution and version of linux.

*disk layout (disk bays, raid controller, etc.).


ext4 doesnt sound like it does what you want. so dont use it...
xfs should handle your situation well. its in the linux kernel, but deployment depends heavily on your disk flexibility. there are also some xfs tunables that will help with your load... out of the box, it wont run particularly well. youll want the right filesystem creation and mount options. the rest depends on your distro, whether youre using a controller, and your particular workload.

a: i guess you answered your own question, the xfs option seems to be the best one (i guess you even get a performance boost). 
the more complex part should be, how to convert ext3/4 into xfs?
if your storage is not a unique physical raid vd (and you didnt make the fs directy on  the blockdevice - mkfs.ext4 /dev/sdb) then i might suggest you into partitioning your fs tree into smaller blocks and mount them accordingly, configuring your software to write to both locations simultaneosly, and splitting the writes if possible.
ex.


*

*/alotofsmallfiles/part1 -> /dev/ext4fs1 

*/alotofsmallfiles/part2 -> /dev/ext4fs2


if splitting the writes is not possible from the application, you could create a cron which moves the files from the ext4 partition into the new xfs every n minutes

a: additional options:


*

*create filesystems on-demand as you need them. partition with lvm instead of attaching your filesystems directly to the mbr. you can mount a fs anywhere in your tree, so you can add a new filesystem whenever and wherever you need it. also, lvm can span portions of multiple disks if you want, meaning physical medium boundaries are less meaningful.

*loopback fses are not a terrible idea, but really why wouldnt you use lvm instead? all the advantages, none of the disadvantages.

*if youre just archiving files (i.e not random-access) then saving them directly into a .tar.gz file is not a bad idea. ive also seen systems where files get staged temporarily to an ssd while the structure is getting built, then get dumped to a tar.gz on a spinning drive for long-term storage.

*xfs is not a bad option, though it has its own difficulties as well. its not quite as forgiving of unclean shutdowns, for example. not that you would expect data loss, but it does sometimes require more hands-on intervention. 
of all these, automatically pushing files into .tar.gz archives is my favorite. it saves you space and inodes, and is just plain tidy. large numbers of small files make filsystems much less performant than expected.

user
In the text, there are one hundred and fourteen "d"s.
assistant