Why does Python only make a copy of the individual element when iterating a list?

Question

I just realized that in Python, if one writes

for i in a:
    i += 1

The elements of the original list a will actually not be affect at all, since the variable i turns out to just be a copy of the original element in a.

In order to modify the original element,

for index, i in enumerate(a):
    a[index] += 1

would be needed.

I was really surprised by this behavior. This seems to be very counterintuitive, seemingly different from other languages and has resulted in errors in my code that I had to debug for a long while today.

I've read Python Tutorial before. Just to be sure, I checked the book again just now, and it doesn't even mention this behavior at all.

What is the reasoning behind this design? Is it expected to be a standard practice in a lot of languages so that the tutorial believes that the readers should get it naturally? In what other languages is the same behavior on iteration present, that I should pay attention to in the future?

Answer 1

I already answered a similar question lately and it's very important to realize that += can have different meanings:

• If the data type implements in-place addition (i.e. has a correctly working __iadd__ function) then the data that i refers to is updated (doesn't matter if it's in a list or somewhere else).

• If the data type doesn't implement an __iadd__ method the i += x statement is just syntactic sugar for i = i + x, so a new value is created and assigned to the variable name i.

• If the data type implements __iadd__ but it does something weird. It could be possible that it's updated ... or not - that depends on what is implemented there.

Pythons integers, floats, strings don't implement __iadd__ so these will not be updated in-place. However other data types like numpy.array or lists implement it and will behave like you expected. So it's not a matter of copy or no-copy when iterating (normally it doesn't do copies for lists and tuples - but that as well depends on the implementation of the containers __iter__ and __getitem__ method!) - it's more a matter of the data type you have stored in your a.

Answer 2

Clarification - terminology

Python does not distinguish between the concepts of reference and pointer. They usually just use the term reference, but if you compare with languages like C++ that do have that distinction - it's much closer to a pointer.

Since the asker clearly comes from C++ background, and since that distinction - which is required for the explanation - does not exist in Python, I've elected to use C++'s terminology, which is:

• Value: Actual data that sits in the memory. void foo(int x); is a signature of a function that receives an integer by value.
• Pointer: A memory address treated as value. Can be deferred to access the memory it points to. void foo(int* x); is a signature of a function that receives an integer by pointer.
• Reference: Sugar around pointers. There is a pointer behind the scenes, but you can only access the deferred value and can not change the address it points to. void foo(int& x); is a signature of a function that receives an integer by reference.

---

What do you mean "different from other languages"? Most languages that I know that support for-each loops are copying the element unless specifically instructed otherwise.

Specifically for Python(though many of these reasons may apply to other languages with similar architectural or philosophical concepts):

1. This behavior may cause bugs for people that are unaware of it, but the alternative behavior may cause bugs even for those who are aware of it. When you assign a variable(i) you usually don't stop and consider all the other variables that would be changed because of it(a). Limiting the scope you are working on is a major factor in preventing spaghetti code, and therefore iteration by copy is usually the default even in languages that support iteration by reference.

2. Python variables are always a single pointer, so it's cheap to iterate by copy - cheaper than iterating by reference, which would require an extra deferring each time you access the value.

3. Python does not have the concept of reference variables like - for example - C++. That is, all variables in Python are actually references, but in the sense that they are pointers - not a behind-the-scenes constat references like C++ type& name arguments. Since this concept does not exist in Python, implementing iteration by reference - let alone making it the default! - will require adding more complexity to the bytecode.

4. Python's for statement works not only on arrays, but on a more general concept of generators. Behind the scenes, Python calls iter on your arrays to get an object which - when you call next on it - either returns the next element or raises a StopIteration. There are several ways to implement generators in Python, and it would have been much harder to implement them for iteration-by-reference.

Answer 3

None of the answers here give you any code to work with to really illustrate why this happens in Python land. And this is fun to look at in a more deep approach so here goes.

The primary reason that this doesn't work as you expect is because in Python, when you write:

i += 1

it is not doing what you think it's doing. Integers are immutable. This can be seen when you look into what the object actually is in Python:

a = 0
print('ID of the first integer:', id(a))
a += 1
print('ID of the first integer +=1:', id(a))

The id function represents a unique and constant value for an object in it's lifetime. Conceptually, it maps loosely to a memory address in C/C++. Running the above code:

ID of the first integer: 140444342529056
ID of the first integer +=1: 140444342529088

This means the first a is no longer the same as the second a, because their id's are different. Effectively they are at different locations in memory.

With an object, however, things work differently. I have overwritten the += operator here:

class CustomInt:
  def __iadd__(self, other):
    # Override += 1 for this class
    self.value = self.value + other.value
    return self

  def __init__(self, v):
    self.value = v

ints = []
for i in range(5):
  int = CustomInt(i)
  print('ID={}, value={}'.format(id(int), i))
  ints.append(int)


for i in ints:
  i += CustomInt(i.value)

print("######")
for i in ints:
  print('ID={}, value={}'.format(id(i), i.value))

Running this results in the following output:

ID=140444284275400, value=0
ID=140444284275120, value=1
ID=140444284275064, value=2
ID=140444284310752, value=3
ID=140444284310864, value=4
######
ID=140444284275400, value=0
ID=140444284275120, value=2
ID=140444284275064, value=4
ID=140444284310752, value=6
ID=140444284310864, value=8

Notice that the id attribute in this case is actually the same for both iterations, even though the value of the object is different (you could also find the id of the int value the object holds, which would be changing as it is mutating - because integers are immutable).

Compare this to when you run the same exercise with an immutable object:

ints_primitives = []
for i in range(5):
  int = i
  ints_primitives.append(int)
  print('ID={}, value={}'.format(id(int), i))

print("######")
for i in ints_primitives:
  i += 1
  print('ID={}, value={}'.format(id(int), i))


print("######")
for i in ints_primitives:
  print('ID={}, value={}'.format(id(i), i))

This outputs:

ID=140023258889248, value=0
ID=140023258889280, value=1
ID=140023258889312, value=2
ID=140023258889344, value=3
ID=140023258889376, value=4
######
ID=140023258889280, value=1
ID=140023258889312, value=2
ID=140023258889344, value=3
ID=140023258889376, value=4
ID=140023258889408, value=5
######
ID=140023258889248, value=0
ID=140023258889280, value=1
ID=140023258889312, value=2
ID=140023258889344, value=3
ID=140023258889376, value=4

A few things here to notice. First, in the loop with the +=, you are no longer adding to the original object. In this case, because ints are among the immutable types in Python, python uses a different id. Also interesting to note that Python uses the same underlying id for multiple variables with the same immutable value:

a = 1999
b = 1999
c = 1999

print('id a:', id(a))
print('id b:', id(b))
print('id c:', id(c))

id a: 139846953372048
id b: 139846953372048
id c: 139846953372048

---

tl;dr - Python has a handful of immutable types, which cause the behavior you see. For all mutable types, your expectation is correct.

Answer 4

@Idan's answer does a good job of explaining why Python does not treat the loop variable as a pointer the way you might in C, but it's worth explaining in more depth how the code snippets unpack, as in Python a lot of simple-seeming bits of code will actually be calls to built in methods. To take your first example

for i in a:
    i += 1

There are two things to unpack: the for _ in _: syntax and the _ += _ syntax. To take the for loop first, like other languages Python has a for-each loop that is essentially syntax sugar for an iterator pattern. In Python, an iterator is an object that defines a .__next__(self) method that returns the current element in the sequence, advances to the next and will raise a StopIteration when there are no more items in the sequence. An Iterable is an object that defines an .__iter__(self) method that returns an iterator.

(N.B.: an Iterator is also an Iterable and returns itself from its .__iter__(self) method.)

Python will usually have an inbuilt function that delegates to the custom double underscore method. So it has iter(o) which resolves to o.__iter__() and next(o) that resolves to o.__next__(). Note these inbuilt functions will often try a reasonable default definition if the method they would delegate to is not defined. For example, len(o) usually resolves to o.__len__() but if that method is not defined it will then try iter(o).__len__().

A for loop is essentially defined in terms of next(), iter() and more basic control structures. In general the code

for i in %EXPR%:
    %LOOP%

will get unpacked to something like

_a_iter = iter(%EXPR%)
while True:
    try:
        i = next(_a_iter)
    except StopIteration:
        break
    %LOOP%

So in this case

for i in a:
    i += 1

gets unpacked to

_a_iter = iter(a) # = a.__iter__()
while True:
    try: 
        i = next(_a_iter) # = _a_iter.__next__()
    except StopIteration:
        break
    i += 1

The other half of this is i += 1. In general %ASSIGN% += %EXPR% gets unpacked to %ASSIGN% = %ASSIGN%.__iadd__(%EXPR%). Here __iadd__(self, other) does inplace addition and returns itself.

(N.B. This is another case where Python will chose an alternative if the main method is not defined. If the object does not implement __iadd__ it will fall back on __add__. It actually does this in this case as int does not implement __iadd__ -- which makes sense because they are immutable and so cannot be modified in place.)

So your code here looks like

_a_iter = iter(a)
while True:
    try:
        i = next(_a_iter)
    except StopIteration:
        break
    i = iadd(i,1)

where we can define

def iadd(o, v):
    try:
        return o.__iadd__(v)
    except AttributeError:
        return o.__add__(v)

There's a bit more going on in your second bit of code. The two new things we need to know are that %ARG%[%KEY%] = %VALUE% gets unpacked to (%ARG%).__setitem__(%KEY%, %VALUE%) and %ARG%[%KEY%] gets unpacked to (%ARG%).__getitem__(%KEY%). Putting this knowledge together we get a[ix] += 1 unpacked to a.__setitem__(ix, a.__getitem__(ix).__add__(1)) (again: __add__ rather than __iadd__ because __iadd__ is not implemented by ints). Our final code looks like:

_a_iter = iter(enumerate(a))
while True:
    try:
        index, i = next(_a_iter)
    except StopIteration:
        break
    a.__setitem__(index, iadd(a.__getitem__(index), 1))

To actually answer your question as to why the first one does not modify the list while the second does, in our first snippet we are getting i from next(_a_iter), which means i will be an int. Since int's cannot be modified in place, i += 1 does nothing to the list. In our second case we are again not modifying the int but are modifying the list by calling __setitem__.

The reason for this whole elaborate exercise is because I think it teaches the following lesson about Python:

1. The price of Python's readability is that it is calling these magic double score methods all the time.
2. Therefore, to have a chance of truly understanding any piece of Python code you have to understand these translations its doing.

The double underscore methods are a hurdle when starting out, but they are essential to backing Python's "runnable pseudocode" reputation. A decent Python programmer will have a thorough understanding of these methods and how they get invoked and will define them wherever it makes good sense to do so.

Edit: @deltab corrected my sloppy use of the term "collection".

Answer 5

+= works differently based on whether the current value is mutable or immutable. This was the main reason that it look a long time for it to get implemented in Python, as Python developers were afraid it would be confusing.

If i is an int, then it cannot be changed since ints are immutable, and thus if the value of i changes then it must necessarily point to another object:

>>> i=3
>>> id(i)
14336296
>>> i+=1
>>> id(i)
14336272   # Other object

However if the left hand side is is mutable, then += can actually change it; like if it it's a list:

>>> i=[]
>>> id(i)
140257231883944
>>> i+=[1]
>>> id(i)
140257231883944  # Still the same object!

In your for loop, i refers to each element of a in turn. If those are integers, then the first case applies, and the result of i += 1 must be that it refers to another integer object. The list a of course still has the same elements it always had.

Answer 6

The loop here is kind of irrelevant. Much like function parameters or arguments, setting up a for loop like that is essentially just fancy-looking assignment.

Integers are immutable. The only way to modify them is by creating a new integer, and assigning it to the same name as the original.

Python's semantics for assignment map directly onto C's (unsurprising given CPython's PyObject* pointers), with the only caveats being that everything is a pointer, and you're not allowed to have double pointers. Consider the following code:

a = 1
b = a
b += 1
print(a)

What happens? It prints 1. Why? It's actually roughly equivalent to the following C code:

i64* a = malloc(sizeof(i64));
*a = 1;
i64* b = a;
i64* tmp = malloc(sizeof(i64));
tmp = *b + 1;
b = tmp;
printf("%d\n", *a);

In the C code, it's obvious that the value of a is completely unaffected.

As for why lists seem to work, the answer is basically just that you're assigning to the same name. Lists are mutable. The identity of the object named a[0] will change, but a[0] is still a valid name. You can check this with the following code:

x = 1
a = [x]
print(a[0] is x)
a[0] += 1
print(a[0] is x)

But, this isn't special for lists. Replace a[0] in that code with y and you get the exact same result.