0ad/source/lib/res/file/file_internal.h

#include "file.h"
#include "file_cache.h"
#include "file_io.h"

#include "file_stats.h"	// must come after file and file_cache

#include "compression.h"
#include "zip.h"
#include "archive.h"

#include "vfs.h"
#include "vfs_path.h"
#include "vfs_mount.h"
#include "vfs_tree.h"

#include "trace.h"
#include "vfs_optimizer.h"

const size_t AIO_SECTOR_SIZE = 512;

// helper routine used by functions that call back to a FileIOCB.
//
// bytes_processed is 0 if return value != { ERR_OK, INFO_CB_CONTINUE }
// note: don't abort if = 0: zip callback may not actually
// output anything if passed very little data.
extern LibError file_io_call_back(const void* block, size_t size,
	FileIOCB cb, uintptr_t ctx, size_t& bytes_processed);
all lib/res/file headers included in file_internal.h split stats into separate file; different call mechanism; add lots of stats (WIP) This was SVN commit r3424. 2006-01-25 08:21:45 +01:00			`#include "file.h"`
			`#include "file_cache.h"`
			`#include "file_io.h"`
v2 of file/res code for thesis: previous zip.* split up into archive, zip and compression. add O(1) lookup via memento. add support for writing Zip archive. simplify IO by having compression module allow queuing input buffers. file: split up into file, file_Io, file_cache. add atom_fn that enables string comparison via pointer compare. (stores all filenames/paths in one place). add statistics gathering. file_cache: landlord algorithm for caching files as well as LRU block cache. file_io: aio code that splits transfers into blocks revamped and made into class. vfs: filelist/trace code moved into new vfs_optimizer. add caching logic to vfs_load. vfs_optimizer: will generate zip archive (to replace current archive builder script). not yet done vfs_tree: major cleanup - now has nice base class rather than union hackery. This was SVN commit r3420. 2006-01-23 21:05:09 +01:00
Cache: implement meat of landlord algorithm and add remove() allocators: add freelist capability to Bucket; add provision for variable XOR fixed size allocs archive: re-tag file buffers if reading uncompressed from archive; improve LFH fixup logic file_cache: add cache line invalidation; lock down pages (readonly) when IO finished file_io: cleanup+docs; properly cut off at EOF without breaking alignment. file_stats: add seek accounting (WIP) vfs_optimizer: also record file_buf_free in the trace. initial implementation of archive builder (WIP) zip: lfh_fixup now more efficient (does not involve buffer manager - instead it grabs LFH from temp blocks) tex: plug FileIOBuf leak. avoid writing to tex.hm because that is a read-only file_buf. This was SVN commit r3428. 2006-01-28 23:19:42 +01:00			`#include "file_stats.h" // must come after file and file_cache`

all lib/res/file headers included in file_internal.h split stats into separate file; different call mechanism; add lots of stats (WIP) This was SVN commit r3424. 2006-01-25 08:21:45 +01:00			`#include "compression.h"`
			`#include "zip.h"`
			`#include "archive.h"`
v2 of file/res code for thesis: previous zip.* split up into archive, zip and compression. add O(1) lookup via memento. add support for writing Zip archive. simplify IO by having compression module allow queuing input buffers. file: split up into file, file_Io, file_cache. add atom_fn that enables string comparison via pointer compare. (stores all filenames/paths in one place). add statistics gathering. file_cache: landlord algorithm for caching files as well as LRU block cache. file_io: aio code that splits transfers into blocks revamped and made into class. vfs: filelist/trace code moved into new vfs_optimizer. add caching logic to vfs_load. vfs_optimizer: will generate zip archive (to replace current archive builder script). not yet done vfs_tree: major cleanup - now has nice base class rather than union hackery. This was SVN commit r3420. 2006-01-23 21:05:09 +01:00
all lib/res/file headers included in file_internal.h split stats into separate file; different call mechanism; add lots of stats (WIP) This was SVN commit r3424. 2006-01-25 08:21:45 +01:00			`#include "vfs.h"`
			`#include "vfs_path.h"`
			`#include "vfs_mount.h"`
			`#include "vfs_tree.h"`
file_cache: fix: add buf/size combo to exact_buf_oracle if file_io rounded size up. trace: split code out of vfs_optimizer (was getting too large) vfs_mount: expose API to unmount all (needed to be able to delete archives after building) vfs_optimizer: safely handle empty connection/node lists; refactored+documented vfs_tree: eliminate 'file in root dir' warning (can happen easily enough) rest: fixes/minor tweaks This was SVN commit r3610. 2006-03-07 06:44:34 +01:00
			`#include "trace.h"`
all lib/res/file headers included in file_internal.h split stats into separate file; different call mechanism; add lots of stats (WIP) This was SVN commit r3424. 2006-01-25 08:21:45 +01:00			`#include "vfs_optimizer.h"`
v2 of file/res code for thesis: previous zip.* split up into archive, zip and compression. add O(1) lookup via memento. add support for writing Zip archive. simplify IO by having compression module allow queuing input buffers. file: split up into file, file_Io, file_cache. add atom_fn that enables string comparison via pointer compare. (stores all filenames/paths in one place). add statistics gathering. file_cache: landlord algorithm for caching files as well as LRU block cache. file_io: aio code that splits transfers into blocks revamped and made into class. vfs: filelist/trace code moved into new vfs_optimizer. add caching logic to vfs_load. vfs_optimizer: will generate zip archive (to replace current archive builder script). not yet done vfs_tree: major cleanup - now has nice base class rather than union hackery. This was SVN commit r3420. 2006-01-23 21:05:09 +01:00
all lib/res/file headers included in file_internal.h split stats into separate file; different call mechanism; add lots of stats (WIP) This was SVN commit r3424. 2006-01-25 08:21:45 +01:00			`const size_t AIO_SECTOR_SIZE = 512;`
automatic archive builder now functional and enabled. - allocators.cpp: fix: make sure all pool fields are reset in pool_free_all; correct matrix self-test - lib: fix self-test - lib_error: give "informational" return codes values > 100 (allows multiplexing with percent indicator) - archive: add clarification to afile_enum; use vfs_load when building archive (fixes caching problem); add support for progressive archive creation - file: allow pp_set_dir's path to end in slash (simplifies other code); add file_exists - file_cache: add stats+sanity checks to allocator; improve ExtantBufMgr find(); add ExactBufOracle to translate padded address returned by Cache to what was actually allocated; relax infinite-loop check; fix file_cache_reset - file_io: refactor callback logic so it can be used by vfs (needed for vfs_load) - vfs: vfs_load now supports calling back (required for archive builder) - vfs_mount: optimize path caching in afile_cb: due to atom_fn system, string compare can be done via pointer compare - vfs_optimizer: support auto-build of archive (progressive); improve should-build-archive check (WIP) - vfs_path: add util functions (also used in screenshot code) - vfs_tree: simplify pathname storage - wposix: mmap bugfix - main: call progressive archive builder; skip reload when doing so (documented) - JSInterface_VFS+scriptglue: add archiveBuilderCancel - ps/Util: use new vfs_path helper function for cleverer screenshot naming (correctly handles some being deleted) This was SVN commit r3578. 2006-03-01 23:31:11 +01:00
			`// helper routine used by functions that call back to a FileIOCB.`
			`//`
			`// bytes_processed is 0 if return value != { ERR_OK, INFO_CB_CONTINUE }`
			`// note: don't abort if = 0: zip callback may not actually`
			`// output anything if passed very little data.`
			`extern LibError file_io_call_back(const void* block, size_t size,`
			`FileIOCB cb, uintptr_t ctx, size_t& bytes_processed);`