# apart of using Numpy/Scipy/Pandas, how can I vectorize code in python

I would like to vectorize some operations in arrays that are not actually available in ndarray or pandas dataframes/series, such as comparing element-wise two arrays/series/frame of similar shape, one of these containing a value and the other one a list. Can I write my own data structure in python which use vectorization? How can I call methods from the general library which vectorize, or use fundamental functions to do this ? Is there no way to vectorize in Python without relying on Fortran/C hence on SciPy/NumPy/Pandas and similar librairies? I don't get why Python by itself would not be able to manage arrays just like C, actually? It is "impossible by conception", or is it just not done because offloads are better?

For example, vectorization is performed when adding two arrays in numpy such as ndarray_1([1,2,3]) and ndarray_2([3,2,6]) which will give ndarray_3([4,4,9]) in one step, and there is no invisible loop, actually all the operations happens in one step in memory. I would like to know how basically code such vectorized operation in python (without using numpy, for my enlightment).

Similarly to what is described above, this would be especially useful to me to know that as I know there are functions exploiting the properties of the special words is and in, such as isin in pandas, which compare a whole series to see whether individual elements are contained in the provided iterator. Unhappily, if I have an array of lists, I have to use a loop to pass over this array to provide sequentially the iterator to be compared in "isin". This is not good at all for my application.

Other use cases would be getting rid of such functions as map and apply, which are disguised for loops with optimizations, to go towards true vectorization. Like applying in one round on a series or frame, element wise, testing of the instance type (isinstance), functions depending on a mathematical formula but also a condition (though, here, I could perform the vectorized mathematical f(x) then in another step apply the boolean formula), and so on. There are tons of use case, indeed I know that I can do some with Numpy/Pandas.

But, First, I can't do all that I want to do (such as finding if an element of a frame is in a list of a comparable frame, element-wise, as stated above, which would be tremendously useful), and it obligates me to curb into steampunk code (which works by some weird transformation of known science, but might not if you look at it closely and skeptically enough, and which is anyway ugly and convoluted) to get away with it. Plus these solutions are not always efficient, and when they are, they only are moderately.

Second, I want to learn to become a better programmer. That means not only relying on the work previously done without understanding it, or relying on hacks or derivative like pushing everything into cython, numba, C and Fortran under the hood. If true vectorized approaches are feasible in python, even if less performant because of the specificities of the language (anyway, we all know that basically, anything out here is slower that C, C++ and Fortran apart maybe a few new languages that are not interpreted but compiled), I would like to learn to know how, as part of improving my skill and understanding of programming.

Hence my question. Thanks for helping me doing it better.

• It really isn't clear what you're asking for here. Python can do anything natively that Numpy can do, but Numpy can do many numeric operations faster since it uses real primitives instead of full objects like Python does. Commented Dec 14, 2017 at 1:32
• I wonder why people feel compelled to negnote when they actually don't understand a simple question which was clearly stated, instead of just asking for precisions. I would not call "doing natively" when it has to rely on another language to perform, even if it is through some of its own libraries. What I want to know is whether it is possible or not to program vectorized operations in pure python and how? Thanks by advance Commented Dec 14, 2017 at 8:21
• Give an example! You are asking if something can be done, and I'm telling you anything can be done in really any language, especially simple numerical stuff. Want to compare two lists element-wise? Use zip in a list comprehension! `[a>b for a, b in zip([1, 2, 3],[3, 2, 1])] == [False, False, True]`. Specifically, check out list comprehensions, generators, and the itertools module in the standard library. Commented Dec 14, 2017 at 8:51
• Ah, you are talking about multithreading. Numpy only actually uses multithreading for a few functions in any case, but even if you have multiple "threads" in python, only one can actually run at a time due to the Global Interpreter Lock. You could use multiprocessing instead to get around this, but the overhead means it's probably only worth it for massive matrices. Commented Dec 14, 2017 at 10:45
• @AndoJurai "when adding two arrays in numpy [...] there is no invisible loop" Yes, there is. Commented Dec 14, 2017 at 11:27

I really hate to burst your bubble after our long discussion in the comments, but your claim:

vectorization is performed when adding two arrays in numpy such as ndarray_1([1,2,3]) and ndarray_2([3,2,6]) which will give ndarray_3([4,4,9]) in one step, and there is no invisible loop, actually all the operations happens in one step in memory.

Is actually not the case at all, there is indeed a "hidden loop" in ALL array based languages, (which you linked to here), and indeed there must be since even the vast majority of newer processes have 8 or fewer threads available, and so in the best instance could not add more than 8 elements of an array at a time (see this stackoverflow question).

You may have heard a lot of talk about processing on GPUs, and this is explicitly for this exact reason. A GPU can process an entire large vector in one go, because GPUs will typically have thousands of cores each with up to ten threads, allowing single pass vector additions for vectors with tens of thousands of elements. Even GPUs, however, will need to split a large enough task into blocks, which effectively is a for loop.

Third party libraries like numba can help you to run native python code (and numpy code!) on a GPU to get the parallelism you want.

Note that even with the advantage of parallel processing, there are overheads involved in moving numbers from CPU to GPU memory, such that for smaller tasks, a simple for loop will still be faster.

• Thanks, that clarifies a lot the problem (I don't hate my mistakes to be pointed out). I actually though that one thread could add in a single pass as many elements as needed as long as the vector stood in adjacent memory blocks. The effect of too many partial knowledges about adjacent topics. So basic solutions are what is quoted above (C++, cython, numba and ufuncs...), as well as multithreading/processing. Another motivation to my question is that I know R is array based, but don't understand why we can't do true arrays in pure python while R do it and is another high level language? Commented Dec 14, 2017 at 11:46
• I honestly think you are confusing data structures from methods on classes. R is a language designed for working with data, while python is a more general purpose language, so it makes sense that R has methods on its arrays which allow simple element-wise summing, but in memory there is nothing magical going on. Commented Dec 14, 2017 at 11:52
• Well, at least based on wikipedia, it happens to exist a difference? And I talked about R but could have talked about Matlab. Python too is made to handle data, it is one of its main purposes. So I wondered why there were no special structure for this (It is widely known R stores its vectors in adjacent blocks in memory, which causes some limitations). Anyway, thanks for your help and valuable information. Commented Dec 14, 2017 at 11:57
• Python also stores objects in its lists in adjacent memory locations laurentluce.com/posts/python-list-implementation. Commented Dec 14, 2017 at 12:05