File sequence


In computing, as well as in non-computing contexts, a file sequence is a well-ordered, collection of files, usually related to each other in some way.
In computing, file sequences should ideally obey some kind of locality of reference principle, so that not only all the files belonging to the same sequence ought to be locally referenced to each other, but they also obey that as much as is their proximity with respect to the ordering relation. Explicit file sequences are, in fact, sequences whose filenames all end with a numeric or alphanumeric tag in the end.
The aforementioned locality of reference usually pertains either to the data, the metadata, or the physical proximity within the storage media they reside in. In the latter acception it is better to speak about file contiguity.

Identification

Every GUI program shows contents of folders by usually ordering its files according to some criteria, mostly related to the files' metadata, like the filename. The criterion is, by default, the alphanumeric ordering of filenames, although some operating systems do that in "smarter" ways than others: for example file1.ext should
ideally be placed before file10.ext, like GNOME Files and Thunar do, whereas, alphanumerically, it comes after.
Other criteria exist, like ordering files by their file type and, if the same type, by either filename or last-access date, and so on.
For this reason, when a file sequence has a more strong locality of reference, particularly when it is related to their actual contents, it is better to highlight this fact by letting their well-ordering induce an alphanumeric ordering of the filenames too. That is the case of explicit file sequences.

Explicit file sequences

Explicit file sequences have the same filename except for the final part, which is a sequence of either numeric, alphanumeric or purely alphabetical characters to force a specific ordering; such sequences should also be ideally located all within the same directory.
In this sense any files sharing the same filename, only differing by the sequence number at the end of the filename, automatically belong to the same file sequence, at least when they are located in the same folder.
It is also part of many naming conventions that number-indexed file sequences containing as many files as to span at most a fixed number of digits, make use of "trailing zeroes" in their filenames so that:
To better explain the latter point, consider that, strictly speaking, file1.ext comes alphanumerically after file100.ext, which is actually the hundredth. By renaming the first file to file001.ext with two trailing zeroes, the problem is universally solved.
Examples of explicit file sequences include: file00000.ext, file00001.ext, file00002.ext,, file02979.ext, and another with a hexadecimal ordering of 256 files tag_00.ext, tag_01.ext,, tag_09.ext, tag_0A.ext,..., tag_0F.ext, tag_10.ext,..., tag_0F.ext,..., tag_FF.ext.
Software and programming conventions usually represent a file sequence as a single virtual file object, whose name is comprehensively written in C-like formatted-string notation to represent where the sequence number is located in the filename and what is its formatting. For the two examples above, that would be filename%05d.ext and tag_%02H.ext, respectively, whereas for the former one, the same convention without trailing zeroes would be filename%5d.ext.
Note, however, that such notation is usually not valid at operating system and command-line interface levels, because the '%' character is neither a valid regular expression nor a universally legal filename character: that notation just stands as a placeholder for the virtual file-like representing the whole explicit file sequence.
Notable software packages acknowledging explicit file sequences as single filesystem objects, rather typical in the Audio/Video post-production industry, are found among products by Autodesk, Quantel, daVinci, DVS, as well as Adobe After Effects.

File scattering

A file sequence located within a mass storage device is said to be contiguous if:
File contiguity is a more practical requirement for file sequences than just their locality of reference, because it is related to the storage medium hosting the whole sequence than to the sequence itself. At the same time, it is a "high-level" feature, because it is not related to the physical and technical details of mass storage itself: particularly, file contiguity is realized in different ways according to the storage device's architecture and actual filesystem structure. At "low level", each file in a contiguous sequence must be placed in contiguous blocks, in spite of reserved areas or special metadata required by the filesystem actually interleaving them.
File contiguity is, in most practical applications, "invisible" at operating-system or user levels, since all the files in a sequence are always available to applications in the same way, regardless of their physical location on the storage device. Indeed, file contiguity may be related to I/O performance when the sequence is to be read or written in the shortest time possible.
In some contexts, data in a file sequence must be accessed in the same order as the file sequence itself; in other contexts, a "random" access to the sequence may be required. In both cases, most professional filesystems provide faster access strategies to contiguous files than non-contiguous ones. Data pre-allocation is crucial for write access, whereas burst read speeds are achievable only for contiguous data.
When a file sequence is not contiguous, it is said to be scattered, since its files are stored in sparse locations on the storage device. File scattering is the process of allocating a file sequence as being uncontiguous. That is often associated with file fragmentation too, where each file is also stored in several, non-contiguous blocks; mechanisms contributing to the former are usually a common cause to the latter too. The act of reducing file scattering by means of allocating or moving files in the same sequence near together on the storage medium is called file descattering.
A few defragmentation strategies and dedicated software are able to both defragment single files and descatter file sequences.

Multimedia file sequences

There are many contexts which explicit file sequences are particularly important in: incremental backups, periodic logs and multimedia files captured or created with a chronological locality of reference. In the latter case, explicit file numbering is extremely important in order to provide both software and end users a way to discern the consequentiality of the contents stored therein. For example, digital cameras and similar devices save all the picture files in the same folder with a final number sequence: it would be very unpractical to choose a filename for each taken shot on the very shooting time, so the camera firmware/software picks one which is perfectly identifiable by its sequence number. With the aid of other metadata, users can later on discern the multimedia contents and re-organize them, if needed.

The Digital Intermediate example

A typical example where explicit file sequences, as well as their contiguity, becomes crucial is in the digital intermediate workflow for motion picture and video industries. In such contexts, video data need to maintain the highest quality and be ready for visualization. Usually video data are acquired from either a digital video camera or a motion picture film scanner and stored into file sequences and need to be post-produced in several steps, including at least editing, conforming and colour-correction. That requires:
Consider that a single frame in a DI project is currently from 9MB to 48MB large, whereas video refresh rate is commonly 24 or 25 frames per second ; any storage required for real-time playing such contents thus needs a minimum overall throughput of 220MB/s to 1.2GB/s, respectively. With those numbers, all the above requirements become strictly mandatory.