Working with multiple python packages in “editable” mode
Like a lot of pythonistas, I prefer doing my python development in virtual environments. Furthermore, I prefer virtualenvwrapper to manage my virtual environments. This ensures isolation of the environment and the packages that the project depends upon and this works extremely well for self contained projects.
Once in the while though, I need to create a dependency to a package that is being worked on in a different virtualenv. In other words, I need to simultaneously work in editable mode on 2 different packages from 2 different virtualenvs.
One of the ways to do this cleanly is to pip install the dependency into your virtualenv. However, each time you update the code for the dependency package, you also need to pip upgrade the package in your virtualenv. This becomes a pain if your dependency is a library that is shared across more than one virtualenv.
The other way to solve this is to also install the dependency in an editable mode, using pip install -e. However, with this approach again, you might end up having to constantly keep the source repo for the dependency package synced across all the virtualenvs that use it.
In both cases, you have multiple copies of the dependency lying around in your virtualenvs. How do you optimize this so that:
- There is just one copy of the dependency package.
- All virtualenvs get the latest changes in the dependency package without having to do anything special.
- Does not interfere or break the isolation of the virtualenv setup (ie: both, the dependency package and the dependent package are still in their own virtualenv)
A good clean way I recently discovered is using add2virtualenv
Perhaps a session capture is much better to explain this than loads of descriptive text:
[steve@lonelap ~]$ mkvirtualenv dependency
New python executable in dependency/bin/python2
Also creating executable in dependency/bin/python
Installing setuptools, pip...done.
[steve@lonelap (dependency)]$ echo 'print("Everyone needs me")' > dependency.py
[steve@lonelap (dependency)]$ deactivate
[steve@lonelap ~]$ mkvirtualenv project
New python executable in project/bin/python2
Also creating executable in project/bin/python
Installing setuptools, pip...done.
[steve@lonelap (project)]$ echo "import dependency" > my_project.py
[steve@lonelap (project)]$ echo "print('I am using dependency from %s' % dependency.__file__)" >> my_project.py
[steve@lonelap (project)]$ python my_project.py
Traceback (most recent call last):
File "my_project.py", line 1, in <module>
import dependency
ImportError: No module named dependency
[steve@lonelap (project)]$ add2virtualenv ~/src/venvs/dependency
[steve@lonelap (project)]$ python my_project.py
Everyone needs me
I am using dependency from /home/steve/src/venvs/dependency/dependency.pyc
[steve@lonelap (project)]$ deactivate
[steve@lonelap ~]$ workon dependency
[steve@lonelap (dependency)]$ echo "print('Dependency is now updated')" >> dependency.py
[steve@lonelap (dependency)]$ deactivate
[steve@lonelap ~]$ workon project
[steve@lonelap (project)]$ python my_project.py
Everyone needs me
Dependency is now updated
I am using dependency from /home/steve/src/venvs/dependency/dependency.py
[steve@lonelap (project)]$
Code reviews instead of whiteboard for interviews
Of late I’ve been thinking a lot about interviews. After being part of this community (or industry if you fancy it) for more than 15 years, I’ve had my share of sitting on both sides of the table. A lot has changed in all these years with the tools we work with as well as the way we approach problems.
However, in all these years not a whole lot has changed in the way we interview (or approach interviewing). Sure, we have better tools and have questioned ourselves about the methodology but at some point we all end up defaulting to focusing on the candidates ability to write good code. This ability though is just a small part of how we work on a daily basis.
The other day I was reviewing a colleague’s patchset and although the code by itself was clean and functional, it was the approach that my colleague had taken which bothered me a bit. So I made my comments and submitted a -1 on the patchset. As is customary when it is easier to just speak about it than reply thru’ the tool (we use gerrit), my colleague walked over to my desk and we spent a good half hour or so discussing the approach. That’s when I got to wondering why don’t we just do more of this during interviews.
- We can quickly weed out people who can’t write code just by testing their ability to comprehend code. If they can’t recognize well known datastuctures or algorithms by looking at the code, they probably won’t be able to write it either.
- Hearing someone review some piece of code gives invaluable insight into just how they think. The colleague I mentioned earlier was new to the company and among other things that we spoke about, we seemed to disagree on whether or not it is a good idea to introduce new patterns/paradigms into existing codebase. I am of the opinion that code should appear as though written by a single author even if it has been worked on by different people. His was that introducing newer patterns improves the quality of the code. While I agreed to an extent with him, I would vote against this if it violates the principle of least surprise [1].
- Hearing someone review some piece of code gives invaluable insight into aspects of their personality. Like for instance their sense of ‘code smell’, their attitude or towards performance trade-offs in a very real practical sense, their ability (/experience) in spotting subtle bugs and just as importantly the manner in which they express their opinions.
- You end up treating the candidates as peers instead of making them feel uneasy or defensive by asking questions they probably could answer when they had just graduated.
So, with these in mind, I am thinking about a scenario where we could possibly use some of our more accessible (isolated) code snapshots which have under gone several revisions during interviews. It might be interesting to see whether we can get the same comments as the rest of the team had made.
Of course, what I’m advocating is not to do away with approaches that judge coding ability but to take a more practical/mature approach to interviewing your peer.
I’m very interested in hearing whether anyone has tried this at their workplace and what other everyday aspects of work do you suppose we can and should incorporate into our interviewing process ?
| [1] | Yeah, I know that this apply to UI rather than code, but IMHO, one ought to consider fellow developers as users of ones code. |
Note: This originally was written as a quick medium post with comments requested at Hacker News
Auto-install missing python modules
I just learned about a neat little python feature introduced in python 3.3 that lets you hook into python’s default import machinery. First a demo:
Ok, so how’s that done ? Well, there are 2 parts to it. Firstly creating the import hook. This is exploiting the new importlib feature for registering a MetaPathFinder class:
The standard lib documentation is very detailed in explaining the how of this mechanism but if you’d like to understand the why and explore other reasons/benefits of being able to hook into the import mechanism you should take some time to read the related PEPs.
The second part to this ensuring that it’s loaded into your environment. The has been done in the demo defining the hook above in the PYTHONSTARTUP file.
If you liked this neat little trick and are interested in other such useful hacks, do check out my custom python startup. Comments and suggestion are always appreciated.
Interview Cake
A while ago I discovered Interview Cake via a Hacker News post and I’m so glad to have found it.
I recommend it to anyone who feels like they are at a disadvantage as programmers because somehow, along the way, they skipped having to learn Algorithms and Data Stuctures and now the subject just seems way too intimidating.
I felt that way for the longest time and I’ve always regretted not having a ^formal computer science^ background. I have tried numerous times to formally study Algorithrms and Data Stuctures from books, videos and other online resources but I often felt handicapped as a programmer for not ever completing the books or videos lectures ...etc
And although, over time I did pick up enough of comp-sci’y knowledge to reason about code and speak intelligently about design, (or sometimes even fake convincingly about those ;-) ), I’ve always tended to linger on the shores of the subject, testily getting my feet wet, never diving deep.
This was primarily because invariably all books that I came across started off being way too technical without a whole lot of practical or motivating pretext for the concepts or approaching the subject with a dry academic sense of purpose. The mental leap necessary to relate what the books taught to getting-shit-done was often times too large ...and I’m (or at least would like to think I am) a pragmatic programmer who wants to solve interesting problems using the tools at hand.
So, although a formal study of the complexity of algorithms made for interesting bedside reading, it never stuck and I’d soon get bored.
Until I saw Interview Cake that is. This site presents problems (typically the kind that you’d face during the technical rounds of an interview for a developer position) – but it isn’t just another codility or project euler, there is a definite sense of purpose to the way the problems are presented, broken down, hints offered and solutions explained.
It makes you want to learn the concepts if you don’t already know the answers and most importantly it is approachable to programmers like me who learned the art of programming by being an apprentice instead of books.
This site is NOT a place to learn the concepts of comp-science but it is a great resource to wheat your appetite. I am now, stoked about ‘deep diving’ in to A&DS and more importantly, I feel like I can remain interested. Also, I feel confident that I’ll be a better programmer due to this.
Learning complex things by solving simple problems
Keeping things organized is not one of my strengths and this is evident by looking at the filesystems on any of the myriad storage devices in my possession.
The folders and files struggle to maintain a sense of perceivable order and intent for being where they are and named as they have been. Beyond the lack of structure and confusing names though, the thing that bothers me the most is the fact that I seem to over time, collect multiple copied of these blasted byte-collections for no good reason.
I’m a byte hoarder and worse still, I can’t seem to organize my hoard.
Although, I do try. Sometimes. Like for instance when I wrote finddup [1]
Now, although finddup started off as being just a tool to allow me to regain some space (and order, in the bit-chaos) on my harddisk, it has turned out to be a wonderful journey of discovery into the concepts of Locality-sensitive hashing and Soundex algorithms. All this because I was dissatisfied with the way I implemented the fuzzy option.
I won’t attempt to explain these concepts here in this post, since firstly, I am not sure I understand them well enough yet and secondly since there already are so many good resources that explain these. However, I do hope to update the fuzzy matching option of finddup to make use of LSH. That way, I can implement it and in the process understand it as well.
What I do what to emphasize here tho’ is this – trying to solve simple problems can teach you a great many new things if you challenge yourself to find better or even different ways to solve those problems.
Remember that future me !
| [1] | Re-wrote actually, not sure how many times I’ve written the simple filename matching and md5sum matching versions of this. |
Hello, World
Although I’ve had the site http://lonetwin.net/ for a while now, where I occasionally (rarely, in fact) blog, I thought of maintaining a relatively more active tech blog which would be different from my usual ranting and gyaan giving at lonetwin.net. Different in the sense that, although I might just rant and give gyaan, it would be more tech related and banal in comparison to serious, thought provoking, deeper (iow, navel gazing) posts o’re there.
Thanks to the graciousness of Vlad Riscutia and various other contributors to Tinkerer, I now have this !
Let’s see where I go with it !