How to Speed up a Python Program 114,000 times.



Optimizations are one thing — making a serious data collection program run 114,000 times faster is another thing entirely.

Leaning on 30+ years of programming experience, David Schachter goes over all the optimizations he made to his (secret) company’s data-collecting program to get such massive performance gains. In doing so, he might be able to teach you a thing or two about optimizing a python program.

Want to learn more about python? Check out more Marakana videos…

, , , ,

28 thoughts on “How to Speed up a Python Program 114,000 times.

  1. I am getting to this vid a bit late. The speaker talks about breaking his application up into threads to get more parallelism on multi-core platforms. This is a laudable goal; however, Python has something called a Global Interpreter Lock (GIL) that limits the effectiveness of multi-threading. I have only done limited Python programming. It would be interesting to investigate the impact of Python's GIL on his application.
    The speaker also makes a statement about the performance of the code as "just good enough" is acceptable (I am paraphrasing here). It is unfortunate that much of the contemporary software development world is so often pressed into this way of doing things. I was taught that, for production code, a certain amount of effort should be dedicated to architecting a design that has goals of efficiency & accuracy with an adherence to time constraints for the overall development phase. I found this video to be interesting and worthwhile.

  2. I don't agree with your bottom line that sexism is still toxic, no matter how you wrap it (like: into comedy). The Big Bang Theory's misogyny is, in my opinion, more reflecting on sexism than anything else. And if it's true that TBBT shows sexism as pathetic, that's exactly not downplaying sexism, but on the contrary making it look what it is: pathetic.
    I cease to understand why some people like to twist and turn everything into "this is bad", even if it is clearly not.

  3. You can use compiler intrinsics to access the AVX and SSE instructions sets – admittedly you're breaking the C standard but you can do it with C and gcc (and in 2012 I believe). So no need for assembly.

  4. About the shared memory and read only data around (12:35), this is unfortunately wrong with most python programs because python will always write “randomly” to memory to maintain its object reference count. Meaning that memory won’t stay shared after a while, and the program will end up using much more memory

  5. Anyone know what program they are using to create this video? I've been wanting to make slide presentation videos like this where the video of the speaker is on the bottom and the slides above.

  6. Good talk. The reason that you got a speed up using Hyperthreaded number of cores is because the CPU can optimize by parallelizing assembly instructions, which are often using different resources in the CPU. 

Leave a Reply

Your email address will not be published. Required fields are marked *