PyFITS v3.1 included an almost complete rewrite of the Header interface. Although the new interface is largely compatible with the old interace (whether due to similarities in the design, or backwards-compatibility support), there are enough differences that a full explanation of the new interface is merited.
The Trac ticket discussing the initial motivation and changes to be made to the Header class is #64. It may be worth reading for some of the background to this work, though this document contains a more complete description of the “final” product (which will continue to evolve).
Prior to 3.1, PyFITS users interacted with FITS headers by way of three different classes: Card, CardList, and Header.
The Card class represents a single header card with a keyword, value, and comment. It also contains all the machinery for parsing FITS header cards, given the 80 character string, or “card image” read from the header.
The CardList class is actually a subclass of Python’s list built-in. It was meant to represent the actual list of cards that make up a header. That is, it represents an ordered list of cards in the physical order that they appear in the header. It supports the usual list methods for inserting and appending new cards into the list. It also supports dict-like keyword access, where cardlist['KEYWORD'] would return the first card in the list with the given keyword.
A lot of the functionality for manipulating headers was actually buried in the CardList class. The Header class was more of a wrapper around CardList that added a little bit of abstraction. It also implemented a partial dict-like interface, though for Headers a keyword lookup returned the header value associated with that keyword, and not the Card object. Though almost every method on the Header class was just performing some operations on the underlying CardList.
The problem is that there were certain things one could only do by directly accessing the CardList, such as look up the comments on a card, or access cards that have duplicate keywords, such as HISTORY. Another long-standing misfeature that slicing a Header object actually returned a CardList object, rather than a new Header. For all but the most simple use cases, working with CardList objects was largely unavoidable.
But it was realized that CardList is really an implementation detail not representing any element of a FITS file distinct from the header itself. Users familiar with the FITS format know what a header is, but it’s not clear how a “card list” is distinct from that, or why operations go through the Header object, while some have to be performed through the CardList.
So the primary goal of this redesign was eliminate the CardList class altogether, and make it possible for users to perform all header manipulations directly through Header objects. It also tries to present headers as similar as possible to more a more familiar data structure–an ordered mapping (or OrderedDict in Python) for ease of use by new users less familiar with the FITS format. Though there are still many added complexities for dealing with the idiosyncracies of the FITS format.
A few old methods on the Header class have been marked as deprecated, either because they have been renamed to a more PEP 8-compliant name, or because have become redundant due to new features. To check if your code is using any deprecated methods or features, run your code with python -Wd. This will output any deprecation warnings to the console.
Two of the most common deprecation warnings related to Headers are for:
The new Header class is designed to work as a drop-in replacement for a dict via duck typing. That is, although it is not a subclass of dict, it implements all the same methods and interfaces. In particular, it is similar to an OrderedDict in that the order of insertions is preserved. However, Header also supports many additional features and behaviors specific to the FITS format. It should also be noted that while the old Header implementation also had a dict-like interface, it did not implement the entire dict interface as the new Header does.
Although the new Header is used like a dict/mapping in most cases, it also supports a list interface. The list-like interace is a bit idiosyncratic in that in some contexts the Header acts like a list of values, in some like a list of keywords, and in a few contexts like a list of Cards. This may be the most difficult aspect of the new design, but there is logic to it.
As with the old Header implementation, integer index access is supported: header[0] returns the value of the first keyword. However, the Header.index() method treats the header as though it’s a list of keywords, and returns the index of a given keyword. For example:
>>> header.index('BITPIX')
2
Header.count() is similar to list.count, and also takes a keyword as its argument:
>>> header.count('HISTORY')
20
A good rule of thumb is that any item access using square brackets [] returns value in the header, whether using keyword or index lookup. Methods like index() and count() that deal with the order and quantity of items in the Header generally work on keywords. Finally, methods like insert() and append() that add new items to the header work on cards.
Aside from the list-like methods, the new Header class works very similarly to the old implementation for most basic use cases and should not present too many surprises. There are differences, however:
As before, the Header() initializer can take a list of Card objects with which to fill the header. However, now any iterable may be used. It is also important to note that any Header method that accepts Card objects can also accept 2-tuples or 3-tuples in place of Cards. That is, either a (keyword, value, comment) tuple or a (keyword, value) tuple (comment is assumed blank) may be used anywhere in place of a Card object. This is even preferred, as it simply involves less typing. For example:
>>> header = pyfits.Header([('A', 1), ('B', 2), ('C', 3, 'A comment')])
>>> header
A = 1
B = 2
C = 3 / A comment
As demonstrated in the previous example, the repr() for a Header, that is the text that is displayed when entering a Header object in the Python console as an expression, shows the header as it would appear in a FITS file. This inserts newlines after each card so that it is easily readable regardless of terminal width. It is not necessary to use print header to view this. Simply entering header displays the header contents as it would appear in the file (sans the END card).
len(header) is now supported (previously it was necessary to do len(header.ascard). This returns the total number of cards in the header, including blank cards, but excluding the END card.
FITS supports having duplicate keywords, although they are generally in error except for commentary keywords like COMMENT and HISTORY. PyFITS now supports reading, updating, and deleting duplicate keywords: Instead of using the keyword by itself, use a (keyword, index) tuple. For example ('HISTORY', 0) represents the first HISTORY card, ('HISTORY', 1) represents the second HISTORY card, and so on. In fact, when a keyword is used by itself, it’s really just shorthand for (keyword, 0). Its is now possible to delete an accidental duplicate like so:
>>> del header[('NAXIS', 1)]
This will remove an accdential duplicate NAXIS card from the header.
Even if there are duplicate keywords, keyword lookups like header['NAXIS'] will always return the value associated with the first copy of that keyword, with one exception: Commentary keywords like COMMENT and HISTORY are expected to have duplicates. So header['HISTORY'], for example, returns the whole sequence of HISTORY values in the correct order. This list of values can be sliced arbitrarily. For example, to view the last 3 history entries in a header:
>>> hdulist[0].header['HISTORY'][-3:]
reference table oref$laf13367o_pct.fits
reference table oref$laf13369o_apt.fits
Heliocentric correction = 16.225 km/s
Subscript assignment can now be used to add new keywords to the header. Just as with a normal dict, header['NAXIS'] = 1 will either update the NAXIS keyword if it already exists, or add a new NAXIS keyword with a value of 1 if it does not exist. In the old interface this would return a KeyError if NAXIS did not exist, and the only way to add a new keyword was through the update() method.
By default, new keywords added in this manner are added to the end of the header, with a few FITS-specific exceptions:
If the header contains extra blank cards at the end, new keywords are added before the blanks.
If the header ends with a list of commentary cards–for example a sequence of HISTORY cards–those are kept at the end, and new keywords are inserted before the commentary cards.
If the keyword is a commentary keyword like COMMENT or HISTORY (or an empty string for blank keywords), a new commentary keyword is always added, and appended to the last commentary keyword of the same type. For example, HISTORY keywords are always placed after the last history keyword:
>>> header = pyfits.Header()
>>> header['COMMENT'] = 'Comment 1'
>>> header['HISTORY'] = 'History 1'
>>> header['COMMENT'] = 'Comment 2'
>>> header['HISTORY'] = 'History 2'
>>> header
COMMENT Comment 1
COMMENT Comment 2
HISTORY History 1
HISTORY History 2
These behaviors represent a sensible default behavior for keyword assignment, and represents the same behavior as update() in the old Header implementation. The default behaviors may still be bypassed through the use of other assignment methods like Header.set() and Header.append() described later.
It is now also possible to assign a value and a comment to a keyword simultaneously using a tuple:
>>> header['NAXIS'] = (2, 'Number of axis')
This will update the value and comment of an existing keyword, or add a new keyword with the given value and comment.
There is a new Header.comments attribute which lists all the comments associated with keywords in the header (not to be confused with COMMENT cards). This allows viewing and updating the comments on specific cards:
>>> header.comments['NAXIS']
Number of axis
>>> header.comments['NAXIS'] = 'Number of axes'
>>> header.comments['NAXIS']
Number of axes
When deleting a keyword from a header, don’t assume that the keyword already exists. In the old Header implementation this would just silently do nothing. For backwards-compatibility it is still okay to delete a non-existent keyword, but a warning will be raised. In the future this will be changed so that trying to delete a non-existent keyword raises a KeyError. This is for consistency with the behavior of Python dicts. So unless you know for certain that a keyword exists before deleting it, it’s best to do something like:
>>> try:
... del header['BITPIX']
... except KeyError:
... pass
Or if you prefer to look before you leap:
>>> if 'BITPIX' in header:
... del header['BITPIX']
del header now supports slices. For example, to delete the last three keywords from a header:
>>> del header[-3:]
Two headers can now be compared for equality–previously no two Header objects were the same. Now they compare as equal if they contain the exact same content. That is, this requires strict equality.
Two headers can now be added with the ‘+’ operator, which returns a copy of the left header extended by the right header with extend(). Assignment addition is also possible.
The Header.update() method used commonly with the old Header API has been renamed to Header.set(). The primary reason for this change is very simple: Header implements the dict interface, which already has a method called update(), but that behaves differently from the old Header.update().
The details of the new update() can be read in the API docs, but it is very similar to dict.update. It also supports backwards compatibility with the old update() by analysis of the arguments passed to it, so existing code will not break immediately. However, this will cause a deprecation warning to be output if they’re enabled. It is best, for starters, to replace all update() calls with set(). Recall, also, that direct assignment is now possible for adding new keywords to a header. So by and large the only reason to prefer using Header.set() is its capability of inserting or moving a keyword to a specific location using the before or after arguments.
Slicing a Header with a slice index returns a new Header containing only those cards contained in the slice. As mentioned earlier, it used to be that slicing a Header returned a card list–something of a misfeature. In general, objects that support slicing ought to return an object of the same type when you slice them.
Likewise, wildcard keywords used to return a CardList object. Now they return a new Header–similarly to a slice. For example:
>>> header['NAXIS*']
returns a new header containing only the NAXIS and NAXISn cards from the original header.
The above may seem like a lot, but the majority of existing code using PyFITS to manipulate headers should not need to be updated, at least not immediately. The most common operations still work the same.
As mentioned above, it would be helpful to run your code with python -Wd to enable deprecation warnings–that should be a good idea of where to look to update your code.
If your code needs to be able to support older versions of PyFITS simultaneously with PyFITS 3.1, things are slightly trickier, but not by much–the deprecated interfaces will not be removed for several more versions because of this.
The first change worth making, which is supported by any PyFITS version in the last several years, is remove any use of Header.has_key() and replace it with keyword in header syntax. It’s worth making this change for any dict as well, since dict.has_key is deprecated. Running the following regular expression over your code may help with most (but not all) cases:
s/([^ ]+)\.has_key\(([^)]+)\)/\2 in \1/
If possible, replace any calls to Header.update() with Header.set() (though don’t bother with this if you need to support older PyFITS versions). Also, if you have any calls to Header.update() that can be replaced with simple subscript assignments (eg. header['NAXIS'] = (2, 'Number of axes')) do that too, if possible.
Find any code that uses header.ascard or header.ascardlist(). First ascertain whether that code really needs to work directly on Card objects. If that is definitely the case, go ahead and replace those with header.cards–that should work without too much fuss. If you do need to support older versions, you may keep using header.ascard for now.
In the off chance that you have any code that slices a header, it’s best to take the result of that and create a new Header object from it. For example:
>>> new_header = pyfits.Header(old_header[2:])
This avoids the problem that in PyFITS <= 3.0 slicing a Header returns a CardList by using the result to initialize a new Header object. This will work in both cases (in PyFITS 3.1, initializing a Header with an existing Header just copies it, a la list).
As mentioned earlier, locate any code that deletes keywords with del, and make sure they either look before they leap (if keyword in header:) or ask forgiveness (try/except KeyError:).
As mentioned above it is not necessary to enter print header to display a header in an interactive Python prompt. Simply entering >>> header by itself is sufficient. Using print usually will not display the header readably, because it does not include line-breaks between the header cards. The reason is that Python has two types of string representations: One is returned when one calls str(header) which happens automatically when you print a variable. In the case of the Header class this actually returns the string value of the header as it is written literally in the FITS file, which includes no line breaks.
The other type of string representation happens when one calls repr(header). The repr of an object is just meant to be a useful string “representation” of the object; in this case the contents of the header but with linebreaks betwen the cards and with the END card and padding trailing padding stripped off. This happens automatically when one enters a variable at the Python prompt by itself without a print call.
The current version of the FITS Standard (3.0) states in section 4.2.1 that trailing spaces in string values in headers are not significant and should be ignored. PyFITS < 3.1 did treat treat trailing spaces as significant. For example if a header contained:
KEYWORD1= ‘Value ‘
then header['KEYWORD1'] would return the string 'Value ' exactly, with the trailing spaces intact. The new Header interface fixes this by automatically stripping trailing spaces, so that header['KEYWORD1'] would return just 'Value'.
There is, however, one convention used by the IRAF ccdmosiac task for representing its TNX World Coordinate System and ZPX World Coordinate System non-standard WCS’ that uses a series of keywords in the form WATj_nnn which store a text description of coefficients for a non-linear distortion projection. It uses its own microformat for listing the coefficients as a string, but the string is long, and thus broken up into several of these WATj_nnn keywords. Correct recombination of these keywords requires treating all whitespace literally. This convention either overlooked or predated the prescribed treatment of whitespace in the FITS standard.
To get around this issue a global variable pyfits.STRIP_HEADER_WHITESPACE was introduced. Temporarily setting pyfits.STRIP_HEADER_WHITESPACE = False before reading keywords affected by this issue will return their values with all trailing whitespace intact.
A future version of PyFITS may be able to detect use of conventions like this contextually and behave according to the convention, but in most cases the default behavior of PyFITS is to behave according to the FITS Standard.