GenericFile

class asdf.generic_io.GenericFile(fd, mode, close=False, uri=None)

Bases: object

Base class for an abstraction layer around a number of different file-like types. Each of its subclasses handles a particular kind of file in the most efficient way possible.

This class should not be instantiated directly, but instead the factory function get_file should be used to get the correct subclass for the given file-like object.

Parameters:

fdfile-like object

The particular kind of file-like object must match the subclass of GenericFile being instantiated.

modestr

Must be "r" (read), "w" (write), or "rw" (read/write).

closebool, optional

When True, close the given fd in the __exit__ method, i.e. at the end of the with block. Should be set to True when this object “owns” the file object. Default: False.

uristr, optional

The file path or URI used to open the file. This is used to resolve relative URIs when the file refers to external sources.

Attributes Summary

block_size
mode	The mode of the file.
uri	The base uri of the file.

Methods Summary

can_memmap()	Returns True if the file supports memmapping.
clear(nbytes)	Write nbytes of zeros.
close()	Close the file.
close_memmap()	Close the memmapped file (if one was mapped with memmap_array)
fast_forward(size)	Move the file position forward by size.
flush()	Flush the internal buffer.
flush_memmap()	Flush any pending writes to the memmapped file (if one was mapped with memmap_array)
is_closed()	Returns True if the underlying file object is closed.
memmap_array(offset, size)	Memmap a chunk of the file into a np.memmap object.
peek([size])	Read bytes of the file without consuming them.
read([size])	Read at most size bytes from the file (less if the read hits EOF before obtaining size bytes).
read_block()	Read a "block" from the file.
read_blocks(size)	Read size bytes of data from the file, one block at a time.
read_into_array(size)	Read a chunk of the file into a uint8 array.
read_until(delimiter, readahead_bytes[, ...])	Reads until a match for a given regular expression is found.
readable()	Returns True if the file can be read from.
reader_until(delimiter, readahead_bytes[, ...])	Returns a readable file-like object that treats the given delimiter as the end-of-file.
seek(offset[, whence])	Set the file's current position.
seek_until(delimiter, readahead_bytes[, ...])	Seeks in the file until a match for a given regular expression is found.
seekable()	Returns True if the file supports random access (seek and tell).
tell()	Return the file's current position, in bytes.
truncate([size])	Truncate the file to the given size.
writable()	Returns True if the file can be written to.
write(content)	Write a string to the file.
write_array(array)	Write array content to the file.

Attributes Documentation

block_size

mode

The mode of the file. Will be 'r', 'w' or 'rw'.

uri

The base uri of the file.

Methods Documentation

can_memmap()

Returns True if the file supports memmapping.

clear(nbytes)

Write nbytes of zeros.

close()

Close the file. The underlying file-object will only be closed if close=True was passed to the constructor.

close_memmap()

Close the memmapped file (if one was mapped with memmap_array)

fast_forward(size)

Move the file position forward by size.

flush()

Flush the internal buffer.

flush_memmap()

Flush any pending writes to the memmapped file (if one was mapped with memmap_array)

is_closed()

Returns True if the underlying file object is closed.

memmap_array(offset, size)

Memmap a chunk of the file into a np.memmap object.

Parameters:

offsetinteger

The offset, in bytes, in the file.

sizeinteger

The size of the data to memmap.

Returns:

arraynp.memmap

peek(size=-1)

Read bytes of the file without consuming them. This method must be implemented by all GenericFile implementations that provide ASDF input (those that aren’t seekable should use a buffer to store peeked bytes).

Parameters:

sizeint

Number of bytes to peek, or -1 to peek all remaining bytes.

read(size=-1)

Read at most size bytes from the file (less if the read hits EOF before obtaining size bytes). If the size argument is negative or omitted, read all data until EOF is reached. The bytes are returned as a bytes object. An empty bytes object is returned when EOF is encountered immediately.

Only available if readable returns True.

read_block()

Read a “block” from the file. For real filesystem files, the block is the size of a native filesystem block.

read_blocks(size)

Read size bytes of data from the file, one block at a time. The result is a generator where each value is a bytes object.

read_into_array(size)

Read a chunk of the file into a uint8 array.

Parameters:

sizeinteger

The size of the data.

Returns:

arraynp.memmap

read_until(delimiter, readahead_bytes, delimiter_name=None, include=True, initial_content=b'', exception=True)

Reads until a match for a given regular expression is found.

Parameters:

delimiterstr

A regular expression.

readahead_bytesint

The number of bytes to read ahead to make sure the delimiter isn’t on a block boundary.

delimiter_namestr, optional

The name of the delimiter. Used in error messages if the delimiter is not found. If not provided, the raw content of delimiter will be used.

includebool, optional

When True, include the delimiter in the result.

initial_contentbytes, optional

Additional content to include at the beginning of the first read.

exceptionbool, optional

If True (default), raise an exception if the end marker isn’t found.

Returns:

contentbytes

The content from the current position in the file, up to the delimiter. Includes the delimiter if include is True.

Raises:

DelimiterNotFoundError

If the delimiter is not found before the end of the file.

readable()

Returns True if the file can be read from.

reader_until(delimiter, readahead_bytes, delimiter_name=None, include=True, initial_content=b'', exception=True)

Returns a readable file-like object that treats the given delimiter as the end-of-file.

Parameters:

delimiterstr

A regular expression.

readahead_bytesint

The number of bytes to read ahead to make sure the delimiter isn’t on a block boundary.

delimiter_namestr, optional

The name of the delimiter. Used in error messages if the delimiter is not found. If not provided, the raw content of delimiter will be used.

includebool, optional

When True, include the delimiter in the result.

initial_contentbytes, optional

Additional content to include at the beginning of the first read.

exceptionbool, optional

If True (default), raise an exception if the end marker isn’t found.

Raises:

DelimiterNotFoundError

If the delimiter is not found before the end of the file.

seek(offset, whence=0)

Set the file’s current position. Only available if seekable returns True.

Parameters:

offsetinteger

Offset, in bytes.

whenceinteger, optional

The whence argument is optional and defaults to SEEK_SET or 0 (absolute file positioning); other values are SEEK_CUR or 1 (seek relative to the current position) and SEEK_END or 2 (seek relative to the file`s end).

seek_until(delimiter, readahead_bytes, delimiter_name=None, include=True, initial_content=b'', exception=True)

Seeks in the file until a match for a given regular expression is found. This is similar to read_until, except the intervening content is not retained.

Parameters:

delimiterstr

A regular expression.

readahead_bytesint

The number of bytes to read ahead to make sure the delimiter isn’t on a block boundary.

delimiter_namestr, optional

The name of the delimiter. Used in error messages if the delimiter is not found. If not provided, the raw content of delimiter will be used.

includebool, optional

When True, include the delimiter in the result.

initial_contentbytes, optional

Additional content to include at the beginning of the first read.

exceptionbool, optional

If True (default), raise an exception if the end marker isn’t found.

Returns:

bool

True if the delimiter was found.

Raises:

DelimiterNotFoundError

If exception is enabled and the delimiter is not found before the end of the file.

seekable()

Returns True if the file supports random access (seek and tell).

tell()

Return the file’s current position, in bytes. Only available in seekable returns True.

truncate(size=None)

Truncate the file to the given size.

writable()

Returns True if the file can be written to.

write(content)

Write a string to the file. There is no return value. Due to buffering, the string may not actually show up in the file until the flush() or close() method is called.

Only available if writable returns True.

write_array(array)

Write array content to the file. Array must be 1D contiguous so that this method can avoid making assumptions about the intended memory layout. Endianness is preserved.

Parameters:

arraynp.ndarray

Must be 1D contiguous.