What’s New In Python 3.4

This article explains the new features in Python 3.4, compared to 3.3.

For full details, see the changelog.

Note

Prerelease users should be aware that this document is currently in draft form. It will be updated substantially as Python 3.4 moves towards release, so it’s worth checking back even after reading earlier versions.

See also

PEP 429 – Python 3.4 Release Schedule

Summary – Release Highlights

New syntax features:

  • No new syntax features were added in Python 3.4.

New expected features for Python implementations:

New library modules:

Significantly Improved Library Modules:

CPython implementation improvements:

Please read on for a comprehensive list of user-facing changes, including many other smaller improvements, CPython optimizations, deprecations, and potential porting issues.

New Expected Features for Python Implementations

PEP 453: Explicit Bootstrapping of PIP in Python Installations

The new ensurepip module (defined in PEP 453) provides a standard cross-platform mechanism to boostrap the pip installer into Python installations and virtual environments.

The venv module and the pyvenv utility make use of this module to make pip readily available in virtual environments. When using the command line interface, pip is installed by default, while for the module API installation of pip must be requested explicitly.

For CPython source builds on POSIX systems, the make install and make altinstall commands bootstrap pip by default. This behaviour can be controlled through configure options, and overridden through Makefile options.

On Windows and Mac OS X, the CPython installers now offer the option to install pip along with CPython itself.

As discussed in the PEP, platform packagers may choose not to install pip by default, as long as the command pip, when invoked, provides clear and simple directions on how to install pip on the platform.

Note

The implementation of PEP 453 is still a work in progress. Refer to issue 19347 for the progress on additional steps:

  • Having the binary installers install pip by default
  • Recommending the use of pip in the “Installing Python Module” documentation.

See also

PEP 453 – Explicit bootstrapping of pip in Python installations
PEP written by Donald Stufft and Nick Coghlan, implemented by Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily.

PEP 446: Make Newly Created File Descriptors Non-Inheritable

PEP 446 makes newly created file descriptors non-inheritable. New functions and methods:

See also

PEP 446 – Make newly created file descriptors non-inheritable
PEP written and implemented by Victor Stinner.

Improvements to Codec Handling

Since it was first introduced, the codecs module has always been intended to operate as a type-neutral dynamic encoding and decoding system. However, its close coupling with the Python text model, especially the type restricted convenience methods on the builtin str, bytes and bytearray types, has historically obscured that fact.

As a key step in clarifying the situation, the codecs.encode() and codecs.decode() convenience functions are now properly documented in Python 2.7, 3.3 and 3.4. These functions have existed in the codecs module (and have been covered by the regression test suite) since Python 2.4, but were previously only discoverable through runtime introspection.

Unlike the convenience methods on str, bytes and bytearray, these convenience functions support arbitrary codecs in both Python 2 and Python 3, rather than being limited to Unicode text encodings (in Python 3) or basestring <-> basestring conversions (in Python 2).

In Python 3.4, the interpreter is able to identify the known non-text encodings provided in the standard library and direct users towards these general purpose convenience functions when appropriate:

>>> b"abcdef".decode("hex")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs

>>> "hello".encode("rot13")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs

In a related change, whenever it is feasible without breaking backwards compatibility, exceptions raised during encoding and decoding operations will be wrapped in a chained exception of the same type that mentions the name of the codec responsible for producing the error:

>>> import codecs

>>> codecs.decode(b"abcdefgh", "hex")
binascii.Error: Non-hexadecimal digit found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found)

>>> codecs.encode("hello", "bz2")
TypeError: 'str' does not support the buffer interface

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface)

Finally, as the examples above show, these improvements have permitted the restoration of the convenience aliases for the non-Unicode codecs that were themselves restored in Python 3.2. This means that encoding binary data to and from its hexadecimal representation (for example) can now be written as:

>>> from codecs import encode, decode
>>> encode(b"hello", "hex")
b'68656c6c6f'
>>> decode(b"68656c6c6f", "hex")
b'hello'

The binary and text transforms provided in the standard library are detailed in Binary Transforms and Text Transforms.

(Contributed by Nick Coghlan in issue 7475, , issue 17827, issue 17828 and issue 19619)

PEP 451: A ModuleSpec Type for the Import System

PEP 451 provides an encapsulation of the information about a module that the import machinery will use to load it (that is, a module specification). This helps simplify both the import implementation and several import-related APIs. The change is also a stepping stone for several future import-related improvements.

The public-facing changes from the PEP are entirely backward-compatible. Furthermore, they should be transparent to everyone but importer authors. Key finder and loader methods have been deprecated, but they will continue working. New importers should use the new methods described in the PEP. Existing importers should be updated to implement the new methods.

Other Language Changes

Some smaller changes made to the core Python language are:

  • Unicode database updated to UCD version 6.3.
  • min() and max() now accept a default argument that can be used to specify the value they return if the iterable they are evaluating has no elements. Contributed by Julian Berman in issue 18111.
  • Module objects are now weakref‘able.
  • Module __file__ attributes (and related values) should now always contain absolute paths by default, with the sole exception of __main__.__file__ when a script has been executed directly using a relative path (Contributed by Brett Cannon in issue 18416).
  • Now all the UTF-* codecs (except UTF-7) reject surrogates during both encoding and decoding unless the surrogatepass error handler is used, with the exception of the UTF-16 decoder that accepts valid surrogate pairs, and the UTF-16 encoder that produces them while encoding non-BMP characters. Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in issue 12892.

New Modules

asyncio

The new asyncio module (defined in PEP 3156) provides a standard pluggable event loop model for Python, providing solid asynchronous IO support in the standard library, and making it easier for other event loop implementations to interoperate with the standard library and each other.

For Python 3.4, this module is considered a provisional API.

See also

PEP 3156 – Asynchronous IO Support Rebooted: the “asyncio” Module
PEP written and implementation led by Guido van Rossum.

ensurepip

The new ensurepip module is the primary infrastructure for the PEP 453 implementation. In the normal course of events end users will not need to interact with this module, but it can be used to manually bootstrap pip if the automated bootstrapping into an installation or virtual environment was declined.

ensurepip includes a bundled copy of pip, up-to-date as of the first release candidate of the release of CPython with which it ships (this applies to both maintenance releases and feature releases). ensurepip does not access the internet. (If the installation has Internet access, it is of course possible to upgrade pip to a release more recent than the bundled pip by using the bundled pip command itself once it is installed.)

The module is named ensurepip because if called when pip is already installed, it does nothing. It also has an --upgrade option that will cause it to install the bundled copy of pip if the existing installed version of pip is older than the bundled copy.

enum

The new enum module (defined in PEP 435) provides a standard implementation of enumeration types, allowing other modules (such as socket) to provide more informative error messages and better debugging support by replacing opaque integer constants with backwards compatible enumeration values.

See also

PEP 435 – Adding an Enum type to the Python standard library
PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman, implemented by Ethan Furman.

pathlib

The new pathlib module offers classes representing filesystem paths with semantics appropriate for different operating systems. Path classes are divided between pure paths, which provide purely computational operations without I/O, and concrete paths, which inherit from pure paths but also provide I/O operations.

For Python 3.4, this module is considered a provisional API.

See also

PEP 428 – The pathlib module – object-oriented filesystem paths
PEP written and implemented by Antoine Pitrou.

selectors

The new selectors module (created as part of implementing PEP 3156) allows high-level and efficient I/O multiplexing, built upon the select module primitives.

statistics

The new statistics module (defined in PEP 450) offers some core statistics functionality directly in the standard library. This module supports calculation of the mean, median, mode, variance and standard deviation of a data series.

See also

PEP 450 – Adding A Statistics Module To The Standard Library
PEP written and implemented by Steven D’Aprano

tracemalloc

The new tracemalloc module (defined in PEP 454) is a debug tool to trace memory blocks allocated by Python. It provides the following information:

  • Traceback where an object was allocated
  • Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks
  • Compute the differences between two snapshots to detect memory leaks

See also

PEP 454 – Add a new tracemalloc module to trace Python memory allocations
PEP written and implemented by Victor Stinner

Improved Modules

abc

New function abc.get_cache_token() can be used to know when to invalidate caches that are affected by changes in the object graph. (Contributed by Łukasz Langa in issue 16832.)

New class ABC has ABCMeta as its meta class. Using ABC as a base class has essentially the same effect as specifying metaclass=abc.ABCMeta, but is simpler to type and easier to read. (Contributed by Bruno Dupuis in issue 16049.)

aifc

The getparams() method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in issue 17818.)

aifc.open() now supports the context manager protocol: when used in a with block, the close() method of the returned object will be called automatically at the end of the block. (Contributed by Serhiy Storchacha in issue 16486.)

argparse

The FileType class now accepts encoding and errors arguments, which are passed through to open(). (Contributed by Lucas Maystre in issue 11175.)

audioop

Added support for 24-bit samples (issue 12866).

Added the byteswap() function to convert big-endian samples to little-endian and vice versa (issue 19641).

base64

The encoding and decoding functions in base64 now accept any bytes-like object in cases where it previously required a bytes or bytearray instance (issue 17839).

colorsys

The number of digits in the coefficients for the RGB — YIQ conversions have been expanded so that they match the FCC NTSC versions. The change in results should be less than 1% and may better match results found elsewhere.

contextlib

The new contextlib.suppress context manager helps to clarify the intent of code that deliberately suppresses exceptions from a single statement. (Contributed by Raymond Hettinger in issue 15806 and Zero Piraeus in issue 19266)

The new contextlib.redirect_stdout() context manager makes it easier for utility scripts to handle inflexible APIs that don’t provide any options to retrieve their output as a string or direct it to somewhere other than sys.stdout. In conjunction with io.StringIO, this context manager is also useful for checking expected output from command line utilities. (Contribute by Raymond Hettinger in issue 15805)

The contextlib documentation has also been updated to include a discussion of the differences between single use, reusable and reentrant context managers.

dis

Functions show_code(), dis(), distb(), and disassemble() now accept a keyword-only file argument that controls where they write their output.

The dis module is now built around an Instruction class that provides object oriented access to the details of each individual bytecode operation.

A new method, get_instructions(), provides an iterator that emits the Instruction stream for a given piece of Python code. Thus it is now possible to write a program that inspects and manipulates a bytecode object in ways different from those provided by the dis module itself. For example:

>>> import dis
>>> for instr in dis.get_instructions(lambda x: x + 1):
...     print(instr.opname)
LOAD_FAST
LOAD_CONST
BINARY_ADD
RETURN_VALUE

The various display tools in the dis module have been rewritten to use these new components.

In addition, a new application-friendly class Bytecode provides an object-oriented API for inspecting bytecode in both in human-readable form and for iterating over instructions. The Bytecode constructor takes the same arguments that get_instruction() does (plus an optional current_offset), and the resulting object can be iterated to produce Instruction objects. But it also has a dis method, equivalent to calling dis on the constructor argument, but returned as a multi-line string:

>>> bytecode = dis.Bytecode(lambda x: x +1, current_offset=3)
>>> for instr in bytecode:
...     print('{} ({})'.format(instr.opname, instr.opcode))
LOAD_FAST (124)
LOAD_CONST (100)
BINARY_ADD (23)
RETURN_VALUE (83)
>>> bytecode.dis().splitlines()       
['  1           0 LOAD_FAST                0 (x)',
 '      -->     3 LOAD_CONST               1 (1)',
 '              6 BINARY_ADD',
 '              7 RETURN_VALUE']

Bytecode also has a class method, from_traceback(), that provides the ability to manipulate a traceback (that is, print(Bytecode.from_traceback(tb).dis()) is equivalent to distb(tb)).

(Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in issue 11816 and Claudiu Popa in issue 17916)

doctest

A new option flag, FAIL_FAST, halts test running as soon as the first failure is detected. (Contributed by R. David Murray and Daniel Urban in issue 16522.)

The doctest command line interface now uses argparse, and has two new options, -o and -f. -o allows doctest options to be specified on the command line, and -f is a shorthand for -o FAIL_FAST (to parallel the similar option supported by the unittest CLI). (Contributed by R. David Murray in issue 11390.)

email

as_string() now accepts a policy argument to override the default policy of the message when generating a string representation of it. This means that as_string can now be used in more circumstances, instead of having to create and use a generator in order to pass formatting parameters to its flatten method.

New method as_bytes() added to produce a bytes representation of the message in a fashion similar to how as_string produces a string representation. It does not accept the maxheaderlen argument, but does accept the unixfrom and policy arguments. The Message __bytes__() method calls it, meaning that bytes(mymsg) will now produce the intuitive result: a bytes object containing the fully formatted message.

(Contributed by R. David Murray in issue 18600.)

A pair of new subclasses of Message have been added, along with a new sub-module, contentmanager. All documentation is currently in the new module, which is being added as part of the new provisional email API. These classes provide a number of new methods that make extracting content from and inserting content into email messages much easier. See the contentmanager documentation for details.

These API additions complete the bulk of the work that was planned as part of the email6 project. The currently provisional API is scheduled to become final in Python 3.5 (possibly with a few minor additions in the area of error handling).

(Contributed by R. David Murray in issue 18891.)

functools

The new partialmethod() descriptor bring partial argument application to descriptors, just as partial() provides for normal callables. The new descriptor also makes it easier to get arbitrary callables (including partial() instances) to behave like normal instance methods when included in a class definition.

(Contributed by Alon Horev and Nick Coghlan in issue 4331)

The new singledispatch() decorator brings support for single-dispatch generic functions to the Python standard library. Where object oriented programming focuses on grouping multiple operations on a common set of data into a class, a generic function focuses on grouping multiple implementations of an operation that allows it to work with different kinds of data.

See also

PEP 443 – Single-dispatch generic functions
PEP written and implemented by Łukasz Langa.

A pure-python version of the partial() function is now in the stdlib; in CPython it is overridden by the C accelerated version, but it is available for other implementations to use. (Contributed by Brian Thorne in issue 12428.)

gc

New function get_stats() returns a list of three per-generation dictionaries containing the collections statistics since interpreter startup. (Contributed by Antoine Pitrou in issue 16351.)

hashlib

New hashlib.pbkdf2_hmac() function. (Contributed by Christian Heimes in issue 18582)

html

Added a new html.unescape() function that converts HTML5 character references to the corresponding Unicode characters. (Contributed by Ezio Melotti in issue 2927)

Added a new convert_charrefs keyword argument to HTMLParser that, when True, automatically converts all character references. For backward-compatibility, its value defaults to False, but it will change to True in future versions, so you are invited to set it explicitly and update your code to use this new feature. (Contributed by Ezio Melotti in issue 13633)

The strict argument of HTMLParser is now deprecated. (Contributed by Ezio Melotti in issue 15114)

http

send_error() now accepts an optional additional exaplain parameter which can be used to provide an extended error description, overriding the hardcoded default if there is one. This extended error description will be formatted using the error_message_format attribute and sent as the body of the error response. (Contributed by Karl Cow in issue 12921.)

importlib

The InspectLoader ABC defines a new method, source_to_code() that accepts source data and a path and returns a code object. The default implementation is equivalent to compile(data, path, 'exec', dont_inherit=True). (Contributed by Eric Snow and Brett Cannon in issue 15627.)

InspectLoader also now has a default implementation for the get_code() method. However, it will normally be desirable to override the default implementation for performance reasons. (Contributed by Brett Cannon in issue 18072.)

inspect

The inspect module now offers a basic command line interface to quickly display source code and other information for modules, classes and functions. (Contributed by Claudiu Popa and Nick Coghlan in issue 18626)

unwrap() makes it easy to unravel wrapper function chains created by functools.wraps() (and any other API that sets the __wrapped__ attribute on a wrapper function). (Contributed by Daniel Urban, Aaron Iles and Nick Coghlan in issue 13266)

As part of the implementation of the new enum module, the inspect module now has substantially better support for custom __dir__ methods and dynamic class attributes provided through metaclasses (Contributed by Ethan Furman in issue 18929 and issue 19030)

logging

The TimedRotatingFileHandler has a new atTime parameter that can be used to specify the time of day when rollover should happen. (Contributed by Ronald Oussoren in issue 9556.)

marshal

The default marshal version has been bumped to 3. The code implementing the new version restores the Python2 behavior of recording only one copy of interned strings and preserving the interning on deserialization, and extends this “one copy” ability to any object type (including handling recursive references). This reduces both the size of .pyc files and the amount of memory a module occupies in memory when it is loaded from a .pyc (or .pyo) file. (Contributed by Kristján Valur Jónsson in issue 16475.)

mmap

mmap objects can now be weakref’ed. (Contributed by Valerie Lambert in issue 4885.)

mock

mock objects now inspect their specification signatures when matching calls, which means an argument can now be matched by either position or name, instead of only by position. (Contributed by Antoine Pitrou in issue 17015.)

mock_open() objects now have readline and readlines methods. (Contributed by Toshio Kuratomi in issue 17467.)

multiprocessing

On Unix, two new start methods (spawn and forkserver) have been added for starting processes using multiprocessing. These make the mixing of processes with threads more robust, and the spawn method matches the semantics that multiprocessing has always used on Windows. (Contributed by Richard Oudkerk in issue 8713).

Also, except when using the old fork start method, child processes will no longer inherit unneeded handles/file descriptors from their parents (part of issue 8713).

multiprocessing now relies on runpy (which implements the -m switch) to initialise __main__ appropriately in child processes when using the spawn or forkserver start methods. This resolves some edge cases where combining multiprocessing, the -m command line switch and explicit relative imports could cause obscure failures in child processes. (Contributed by Nick Coghlan in issue 19946)

operator

There is now a pure-python version of the operator module available for reference and for use by alternate implementations of Python. (Contributed by Zachary Ware in issue 16694.)

os

New functions to get and set the inheritable flag of a file descriptors or a Windows handle:

The os module now provides a cpu_count() function, analogous to the multiprocessing.cpu_count() function (which is now implemented in terms of the new os function). (Contributed by Trent Nelson, Yogesh Chaudhari, Victor Stinner, and Charles-François Natali in issue 17914.)

pdb

The print command has been removed from pdb, restoring access to the print function.

Rationale: Python2’s pdb did not have a print command; instead, entering print executed the print statement. In Python3 print was mistakenly made an alias for the pdb p command. p, however, prints the repr of its argument, not the str like the Python2 print command did. Worse, the Python3 pdb print command shadowed the Python3 print function, making it inaccessible at the pdb prompt.

(Contributed by Connor Osborn in issue 18764.)

pickle

protocol 4

pickle now supports (but does not use by default) a new pickle protocol, protocol 4. This new protocol addresses a number of issues that were present in previous protocols, such as the serialization of nested classes, very large strings and containers, or classes whose __new__() method takes keyword-only arguments. It also provides some efficiency improvements.

See also

PEP 3154 – Pickle protocol 4
PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti.

poplib

Two new methods have been added to poplib: capa(), which returns the list of capabilities advertised by the POP server, and stls(), which switches a clear-text POP3 session into an encrypted POP3 session if the POP server supports it. (Contributed by Lorenzo Catucci in issue 4473.)

pprint

The pprint module now supports compact mode for formatting long sequences (issue 19132).

Long strings are now wrapped using Python’s normal line continuation syntax (Contributed by Antoine Pitrou in issue 17150.)

pty

pty.spawn() now returns the status value from os.waitpid() on the child process, instead of None. (Contributed by Gregory P. Smith.)

pydoc

While significant changes have not been made to pydoc directly, its handling of custom __dir__ methods and various descriptor behaviours has been improved substantially by the underlying changes in the inspect module.

re

Added re.fullmatch() function and regex.fullmatch() method, which anchor the pattern at both ends of the string to match. (Contributed by Matthew Barnett in issue 16203.)

The repr of regex objects now includes the pattern and the flags; the repr of match objects now includes the start, end, and the part of the string that matched. (Contributed by Serhiy Storchaka in issue 13592 and issue 17087.)

resource

New resource.prlimit() function and Linux specific constants. (Contributed by Christian Heimes in issue 16595 and issue 19324.)

select

epoll objects now support the context management protocol. When used in a with statement, the close() method will be called automatically at the end of the block. (Contributed by Serhiy Storchaka in issue 16488.)

shelve

Shelf instances may now be used in with statements, and will be automatically closed at the end of the with block. (Contributed by Filip Gruszczyński in issue 13896.)

smtplib

SMTPException is now a subclass of OSError, which allows both socket level errors and SMTP protocol level errors to be caught in one try/except statement by code that only cares whether or not an error occurred. (issue 2118).

socket

The socket module now supports the CAN_BCM protocol on platforms that support it. (Contributed by Brian Thorne in issue 15359.)

Socket objects have new methods to get or set their inheritable flag, get_inheritable() and set_inheritable().

The socket.AF_* and socket.SOCK_* constants are now enumeration values using the new enum module. This allows meaningful names to be printed during debugging, instead of integer “magic numbers”.

The AF_LINK constant is now available on BSD and OSX.

sqlite3

A new boolean parameter, uri, to the connect() function can be used to indicate that the database parameter is a uri (see the SQLite URI documentation). (Contributed by poq in issue 13773.)

ssl

PROTOCOL_TLSv1_1 and PROTOCOL_TLSv1_2 (TLSv1.1 and TLSv1.2 support) have been added; support for these protocols is only available if Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orrù and Antoine Pitrou in issue 16692)

New diagnostic functions get_default_verify_paths(), cert_store_stats() and get_ca_certs() (Contributed by Christian Heimes in issue 18143 and issue 18147)

Add ssl.enum_cert_store() to retrieve certificates and CRL from Windows’ cert store. (Contributed by Christian Heimes in issue 17134.)

Support for server-side SNI using the new ssl.SSLContext.set_servername_callback() method. (Contributed by Daniel Black in issue 8109.)

stat

The stat module is now backed by a C implementation in _stat. A C implementation is required as most of the values aren’t standardized and platform-dependent. (Contributed by Christian Heimes in issue 11016.)

The module supports new file types: door, event port and whiteout.

struct

struct now supports the streamed unpacking of a buffer containing repeated instances of a given format of data. Both a module level iter_unpack function and a struct.Struct.iter_unpack() method on compiled formats have been added. (Contributed by Antoine Pitrou in issue 17804.)

subprocess

check_output() now accepts an input argument that can be used to provide the contents of stdin for the command that is run. (Contributed by Zack Weinberg in issue 16624.)

sunau

The getparams() method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in issue 18901.)

sunau.open() now supports the context manager protocol (issue 18878).

sys

New function sys.getallocatedblocks() returns the current number of blocks allocated by the interpreter (in CPython with the default --with-pymalloc setting, this is allocations made through the PyObject_Malloc() API). This can be useful for tracking memory leaks, especially if automated via a test suite. (Contributed by Antoine Pitrou in issue 13390.)

traceback

A new traceback.clear_frames() function takes a traceback object and clears the local variables in all of the frames it references, reducing the amount of memory consumed (issue 1565525).

urllib

urllib.request now supports data: URLs via the DataHandler class. (Contributed by Mathias Panzenböck in issue 16423.)

unittest

The TestCase class has a new method, subTest(), that produces a context manager whose with block becomes a “sub-test”. This context manager allows a test method to dynamically generate subtests by, say, calling the subTest context manager inside a loop. A single test method can thereby produce an indefinite number of separately-identified and separately-counted tests, all of which will run even if one or more of them fail. For example:

class NumbersTest(unittest.TestCase):
    def test_even(self):
        for i in range(6):
            with self.subTest(i=1):
                self.assertEqual(i % 2, 0)

will result in six subtests, each identified in the unittest verbose output with a label consisting of the variable name i and a particular value for that variable (i=0, i=1, etc). See Distinguishing test iterations using subtests for the full version of this example. (Contributed by Antoine Pitrou in issue 16997.)

unittest.main() now accepts an iterable of test names for defaultTest, where previously it only accepted a single test name as a string. (Contributed by Jyrki Pulliainen in issue 15132.)

If SkipTest is raised during test discovery (that is, at the module level in the test file), it is now reported as a skip instead of an error. (Contributed by Zach Ware in issue 16935.)

discover() now sorts the discovered files to provide consistent test ordering. (Contributed by Martin Melin and Jeff Ramnani in issue 16709.)

venv

venv now includes activation scripts for the csh and fish shells (Contributed by Andrew Svetlov in issue 15417.)

wave

The getparams() method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in issue 17487.)

wave.open() now supports the context manager protocol. (Contributed by Claudiu Popa in issue 17616.)

weakref

New WeakMethod class simulates weak references to bound methods. (Contributed by Antoine Pitrou in issue 14631.)

New finalize class makes it possible to register a callback to be invoked when an object is garbage collected, without needing to carefully manage the lifecycle of the weak reference itself. (Contributed by Richard Oudkerk in issue 15528)

The callback, if any, associated with a ref is now exposed via the __callback__ attribute. (Contributed by Mark Dickinson in issue 17643.)

xml.etree

A new parser, XMLPullParser, allows a non-blocking applications to parse XML documents. An example can be seen at Pull API for non-blocking parsing. (Contributed by Antoine Pitrou in issue 17741.)

The xml.etree.ElementTree tostring() and tostringlist() functions, and the ElementTree write() method, now have a short_empty_elements keyword-only parameter providing control over whether elements with no content are written in abbreviated (<tag />) or expanded (<tag></tag>) form. (Contributed by Ariel Poliak and Serhiy Storchaka in issue 14377.)

zipfile.PyZipfile

Add a filter function to ignore some packages (tests for instance), writepy(). (Contributed by Christian Tismer in issue 19274.)

CPython Implementation Changes

PEP 445: Customization of CPython Memory Allocators

PEP 445 adds new C level interfaces to customize memory allocation in the CPython interpreter.

See also

PEP 445 – Add new APIs to customize Python memory allocators
PEP written and implemented by Victor Stinner.

PEP 442: Safe Object Finalization

PEP 442 removes the current limitations and quirks of object finalization in CPython. With it, objects with __del__() methods, as well as generators with finally clauses, can be finalized when they are part of a reference cycle.

As part of this change, module globals are no longer forcibly set to None during interpreter shutdown in most cases, instead relying on the normal operation of the cyclic garbage collector. This avoids a whole class of interpreter-shutdown-time errors, usually involving __del__ methods, that have plagued Python since the cyclic GC was first introduced.

See also

PEP 442 – Safe object finalization
PEP written and implemented by Antoine Pitrou.

PEP 456: Secure and Interchangeable Hash Algorithm

PEP 456 follows up on earlier security fix work done on Python’s hash algorithm to address certain DOS attacks to which public facing APIs backed by dictionary lookups may be subject. (See issue 14621 for the start of the current round of improvements.) The PEP unifies CPython’s hash code to make it easier for a packager to substitute a different hash algorithm, and switches Python’s default implementation to a SipHash implementation on platforms that have a 64 bit data type. Any performance differences in comparison with the older FNV algorithm are trivial.

The PEP adds additional fields to the sys.hash_info() struct sequence to describe the hash algorithm in use by the currently executing binary. Otherwise, the PEP does not alter any existing CPython APIs.

PEP 436: Argument Clinic

“Argument Clinic” (PEP 436) is now part of the CPython build process and can be used to simplify the process of defining and maintaining accurate signatures for builtins and standard library extension modules implemented in C.

Note

The Argument Clinic PEP is not fully up to date with the state of the implementation. This has been deemed acceptable by the release manager and core development team in this case, as Argument Clinic will not be made available as a public API for third party use in Python 3.4.

See also

PEP 436 – The Argument Clinic DSL
PEP written and implemented by Larry Hastings.

Other Build and C API Changes

  • The new Py_SetStandardStreamEncoding() pre-initialization API allows applications embedding the CPython interpreter to reliably force a particular encoding and error handler for the standard streams (Contributed by Bastien Montagne and Nick Coghlan in issue 16129)
  • Most Python C APIs that don’t mutate string arguments are now correctly marked as accepting const char * rather than char * (Contributed by Serhiy Storchaka in issue 1772673).
  • New shell version of python-config; can be used even when a python interpreter is not available (for example, in cross compilation scenarios).

Other Improvements

  • Tab-completion is now enabled by default in the interactive interpreter. (Contributed by Antoine Pitrou and Éric Araujo in issue 5845.)
  • Invoking the Python interpreter with --version now outputs the version to standard output instead of standard error (issue 18338). Similar changes were made to argparse (issue 18920) and other modules that have script-like invocation capabilities (issue 18922).
  • The CPython Windows installer now adds .py to the PATHEXT variable when extensions are registered, allowing users to run a python script at the windows command prompt by just typing its name without the .py extension. (Contributed by Paul Moore in issue 18569.)
  • A new make target coverage-report will build python, run the test suite, and generate an HTML coverage report for the C codebase using gcov and lcov.
  • The -R option to the python regression test suite now also checks for memory allocation leaks, using sys.getallocatedblocks(). (Contributed by Antoine Pitrou in issue 13390).

Significant Optimizations

  • The UTF-32 decoder is now 3x to 4x faster. (Contributed by Serhiy Storchaka in issue 14625.)
  • The cost of hash collisions for sets is now reduced. Each hash table probe now checks a series of consecutive, adjacent key/hash pairs before continuing to make random probes through the hash table. This exploits cache locality to make collision resolution less expensive. The collision resolution scheme can be described as a hybrid of linear probing and open addressing. The number of additional linear probes defaults to nine. This can be changed at compile-time by defining LINEAR_PROBES to be any value. Set LINEAR_PROBES=0 to turn-off linear probing entirely. (Contributed by Raymond Hettinger in issue 18771.)
  • The interpreter starts about 30% faster. A couple of measures lead to the speedup. The interpreter loads fewer modules on startup, e.g. the re, collections and locale modules and their dependencies are no longer imported by default. The marshal module has been improved to load compiled Python code faster. (Contributed by Antoine Pitrou, Christian Heimes and Victor Stinner in issue 19219, issue 19218, issue 19209, issue 19205 and issue 9548)
  • bz2.BZ2File is now as fast or faster than the Python2 version for most cases. lzma.LZMAFile has also been optimized. (Contributed by Serhiy Storchaka and Nadeem Vawda in issue 16034.)
  • random.getrandbits() is 20%-40% faster for small integers (the most common use case). (Contributed by Serhiy Storchaka in issue 16674).
  • By taking advantage of the new storage format for strings, pickling of strings is now significantly faster. (Contributed by Victor Stinner and Antoine Pitrou in issue 15596.)
  • A performance issue in io.FileIO.readall() has been solved. This particularly affects Windows, and significantly speeds up the case of piping significant amounts of data through subprocess. (Contributed by Richard Oudkerk in issue 15758.)

Deprecated

This section covers various APIs and other features that have been deprecated in Python 3.4, and will be removed in Python 3.5 or later. In most (but not all) cases, using the deprecated APIs will produce a DeprecationWarning when the interpreter is run with deprecation warnings enabled (for example, by using -Wd).

Deprecated Python Modules, Functions and Methods

  • difflib.SequenceMatcher.isbjunk() and difflib.SequenceMatcher.isbpopular() were removed: use x in sm.bjunk and x in sm.bpopular, where sm is a SequenceMatcher object.
  • importlib.util.module_for_loader() is pending deprecation. Using importlib.util.module_to_load() and importlib.abc.Loader.init_module_attrs() allows subclasses of a loader to more easily customize module loading.
  • The imp module is pending deprecation. To keep compatibility with Python 2/3 code bases, the module’s removal is currently not scheduled.
  • The formatter module is pending deprecation and is slated for removal in Python 3.6.
  • MD5 as default digestmod for hmac is deprecated. Python 3.6 will require an explicit digest name or constructor as digestmod argument.
  • The internal Netrc class in the ftplib module has been documented as deprecated in its docstring for quite some time. It now emits a DeprecationWarning and will be removed completely in Python 3.5.

Deprecated Functions and Types in the C API

XXX: None so far

Deprecated Features

  • Running IDLE with the -n flag (no subprocess) is deprecated. However, the feature will not be removed until issue 18823 is resolved.
  • The site module adding a “site-python” directory to sys.path, if it exists, is deprecated (issue 19375).

Removed

Operating Systems No Longer Supported

Support for the following operating systems has been removed from the source and build tools:

API and Feature Removals

The following obsolete and previously deprecated APIs and features have been removed:

  • The unmaintained Misc/TextMate and Misc/vim directories have been removed (see the devguide for suggestions on what to use instead).
  • The SO makefile macro is removed (it was replaced by the SHLIB_SUFFIX and EXT_SUFFIX macros) (issue 16754).
  • The PyThreadState.tick_counter field has been removed; its value has been meaningless since Python 3.2, when the “new GIL” was introduced.
  • PyLoader and PyPycLoader have been removed from importlib. (Contributed by Taras Lyapun in issue 15641.)
  • The strict argument to HTTPConnection and HTTPSConnection has been removed. HTTP 0.9-style “Simple Responses” are no longer supported.
  • The deprecated urllib.request.Request getter and setter methods add_data, has_data, get_data, get_type, get_host, get_selector, set_proxy, get_origin_req_host, and is_unverifiable have been removed (use direct attribute access instead).
  • Support for loading the deprecated TYPE_INT64 has been removed from marshal. (Contributed by Dan Riti in issue 15480.)

Code Cleanups

  • The unused and undocumented internal Scanner class has been removed from the pydoc module.

Porting to Python 3.4

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in the Python API

  • The ABCs defined in importlib.abc now either raise the appropriate exception or return a default value instead of raising NotImplementedError blindly. This will only affect code calling super() and falling through all the way to the ABCs. For compatibility, catch both NotImplementedError or the appropriate exception as needed.
  • The module type now initializes the __package__ and __loader__ attributes to None by default. To determine if these attributes were set in a backwards-compatible fashion, use e.g. getattr(module, '__loader__', None) is not None.
  • importlib.util.module_for_loader() now sets __loader__ and __package__ unconditionally to properly support reloading. If this is not desired then you will need to set these attributes manually. You can use importlib.util.module_to_load() for module management.
  • Import now resets relevant attributes (e.g. __name__, __loader__, __package__, __file__, __cached__) unconditionally when reloading.
  • Frozen packages no longer set __path__ to a list containing the package name but an empty list instead. Determing if a module is a package should be done using hasattr(module, '__path__').
  • py_compile.compile() now raises FileExistsError if the file path it would write to is a symlink or a non-regular file. This is to act as a warning that import will overwrite those files with a regular file regardless of what type of file path they were originally.
  • importlib.abc.SourceLoader.get_source() no longer raises ImportError when the source code being loaded triggers a SyntaxError or UnicodeDecodeError. As ImportError is meant to be raised only when source code cannot be found but it should, it was felt to be over-reaching/overloading of that meaning when the source code is found but improperly structured. If you were catching ImportError before and wish to continue to ignore syntax or decoding issues, catch all three exceptions now.
  • functools.update_wrapper() and functools.wraps() now correctly set the __wrapped__ attribute to the function being wrapper, even if that function also had its __wrapped__ attribute set. This means __wrapped__ attributes now correctly link a stack of decorated functions rather than every __wrapped__ attribute in the chain referring to the innermost function. Introspection libraries that assumed the previous behaviour was intentional can use inspect.unwrap() to access the first function in the chain that has no __wrapped__ attribute.
  • importlib.machinery.PathFinder now passes on the current working directory to objects in sys.path_hooks for the empty string. This results in sys.path_importer_cache never containing '', thus iterating through sys.path_importer_cache based on sys.path will not find all keys. A module’s __file__ when imported in the current working directory will also now have an absolute path, including when using -m with the interpreter (this does not influence when the path to a file is specified on the command-line).
  • The removal of the strict argument to HTTPConnection and HTTPSConnection changes the meaning of the remaining arguments if you are specifying them positionally rather than by keyword. If you’ve been paying attention to deprecation warnings your code should already be specifying any additional arguments via keywords.
  • Strings between from __future__ import ... statements now always raise a SyntaxError. Previously if there was no leading docstring, an interstitial string would sometimes be ignored. This brings CPython into compliance with the language spec; Jython and PyPy already were. (issue 17434).
  • ssl.SSLSocket.getpeercert() and ssl.SSLSocket.do_handshake() now raise an OSError with ENOTCONN when the SSLSocket is not connected, instead of the previous behavior of raising an AttributError. In addition, getpeercert() will raise a ValueError if the handshake has not yet been done.
  • base64.b32decode() now raises a binascii.Error when the input string contains non-b32-alphabet characters, instead of a TypeError. This particular TypeError was missed when the other TypeErrors were converted. (Contributed by Serhiy Storchaka in issue 18011.) Note: this change was also inadvertently applied in Python 3.3.3.

Changes in the C API

Table Of Contents

Previous topic

What’s New in Python

Next topic

What’s New In Python 3.3

This Page