Author Topic: eggs in site-packages not imported with latest Anaconda  (Read 2209 times)

aavanian

  • Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
    • View Profile
eggs in site-packages not imported with latest Anaconda
« on: September 23, 2015, 02:53:06 PM »
Hi, I'm having a related issue.

I'm currently building a package that is installed with 'python setup.py develop'. But it seems the .pth files aren't processed when building sys.path.
I am using anaconda, with the root environment. Importing true installs (like numpy, pandas) is working well.
I actually discovered the issue after extracting a series of utility functions in a new sub-package of my project.

A terminal will have a sys.path like this:
Quote
['',
 'c:\\users\\aavanian\\documents\\projects\\val_tool',
 'C:\\Anaconda3\\python34.zip',
 'C:\\Anaconda3\\DLLs',
 'C:\\Anaconda3\\lib',
 'C:\\Anaconda3',
 'C:\\Anaconda3\\lib\\site-packages',
 'C:\\Anaconda3\\lib\\site-packages\\Sphinx-1.3.1-py3.4.egg',
 'C:\\Anaconda3\\lib\\site-packages\\cryptography-0.9.1-py3.4-win-amd64.egg',
 'C:\\Anaconda3\\lib\\site-packages\\win32',
 'C:\\Anaconda3\\lib\\site-packages\\win32\\lib',
 'C:\\Anaconda3\\lib\\site-packages\\Pythonwin',
 'C:\\Anaconda3\\lib\\site-packages\\setuptools-18.1-py3.4.egg']

Whereas pyxll has this:
Quote
2015-09-23 21:40:57,119 - INFO : ['C:\\Anaconda3\\Lib\\site-packages',
2015-09-23 21:40:57,119 - INFO :  'C:\\Anaconda3\\Lib\\site-packages\\win32',
2015-09-23 21:40:57,119 - INFO :  'C:\\Anaconda3\\Lib\\site-packages\\win32\\lib',
2015-09-23 21:40:57,119 - INFO :  'C:\\Anaconda3\\Lib\\site-packages\\win32\\Pythonwin',
2015-09-23 21:40:57,119 - INFO :  'C:\\Users\\aavanian\\Documents\\Projects\\val_tool\\val_tool\\utils',
2015-09-23 21:40:57,119 - INFO :  '',
2015-09-23 21:40:57,119 - INFO :  'C:\\Anaconda3\\python34.zip',
2015-09-23 21:40:57,119 - INFO :  'C:\\Anaconda3\\Lib',
2015-09-23 21:40:57,120 - INFO :  'C:\\Anaconda3\\DLLs',
2015-09-23 21:40:57,120 - INFO :  '',
2015-09-23 21:40:57,120 - INFO :  'C:\\Program Files\\Microsoft Office 15\\root\\office15',
2015-09-23 21:40:57,120 - INFO :  'C:\\Anaconda3']
Where the first five lines come from pyxll.cfg.

I dig into this as I can't seem to import other modules from the val_tool package I'm building (except if the module I'm trying to import is in the same sub-package than the module I'm importing it from). Even adding paths to pyxll.cfg doesn't help.

Regards,
Alexandre

Tony

  • Administrator
  • Hero Member
  • *****
  • Posts: 581
  • Karma: +13/-1
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #1 on: September 23, 2015, 03:07:22 PM »
Hi Alexandre,

the .pth files should get processed when site.py is imported, and that's what added the extra paths the the python path.

You can check what site.py file is imported by setting the environment variable PYXLL_DEBUG_LOG=c:\temp\pyxll-debug-log.txt (for example). That will print out some more debug information before Python has been fully initialized, including where site.py is being imported from.

You can check what site.py is used in your normal python prompt by importing it and printing it, i.e.

Code: [Select]
import site
print(site.__file__)

If your package in documents\projects\val_tool, or documents\projects\val_tool\utils? You seem to have one in your normal python path and the other in your pyxll config. Maybe that's why you can't import it even though it's in your config file?

Best regards,
Tony

aavanian

  • Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #2 on: September 23, 2015, 03:26:24 PM »
I actually have been using the PYXLL_DEBUG_LOG feature but the log doesn't show anything relevant (as far as I can see, I may miss it though). It's a bunch of environment variables, only a few of them being related to python or pyxll. Please let me know the key or variable I should be looking for.

As for my project structure, it's the following:
Quote
val_tool/ (actually C:/users/aavanian/documents/projects/val_tool/)
    val_tool/
        utils/
            settings.py
            ...
        vtio/
            ...
        est/
            ...
    setup.py
    ...

I added the path to utils/ in pyxll.cfg only to be able to print the sys.path variable.

My ideal target is not to have to add anything (or manage it through the pyxll-utils extension loader)

What I have been doing before is to add the vtio/ and est/ paths in pyxll.cfg which are the sub-packages that contains the modules exposing functions/macros to Excel. But when doing that, these modules can only import from the same sub-package, not from the utils/ sub-package. Adding the path to utils/ doesn't help (and would only be a band-aid outside of development).

Regards
Alexandre

PS: Thanks for the quick answers!

Tony

  • Administrator
  • Hero Member
  • *****
  • Posts: 581
  • Karma: +13/-1
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #3 on: September 23, 2015, 03:37:00 PM »
Hi Alexandre,

you're looking for something like this:

2015-09-23 13:54:06,129 - DEBUG : Looking for module 'site'...
2015-09-23 13:54:06,131 - DEBUG : Imported site (C:\xxxx\Lib\site.py)

You may find it in the main log file instead of the pyxll-debug log file (but you still need the environment variable set). Towards the end of the pyxll-debug log file it should have a line about redirecting to the other log once it's read the config.

Best regards,
Tony

aavanian

  • Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #4 on: September 23, 2015, 03:41:42 PM »
Ah indeed, here it is:
Quote
2015-09-23 21:40:56,962 - DEBUG : Looking for module 'site'...
2015-09-23 21:40:56,971 - DEBUG : Imported site (C:\Anaconda3\Lib\site.py)
2015-09-23 21:40:56,993 - DEBUG : ---- pyxll module is being imported ----
2015-09-23 21:40:57,010 - DEBUG :   threading imported: C:\Anaconda3\Lib\threading.py
2015-09-23 21:40:57,019 - DEBUG :   glob imported: C:\Anaconda3\Lib\glob.py
2015-09-23 21:40:57,022 - DEBUG :   re imported: C:\Anaconda3\Lib\re.py
2015-09-23 21:40:57,025 - DEBUG :   sys imported: None
2015-09-23 21:40:57,027 - DEBUG :   os imported: C:\Anaconda3\Lib\os.py
2015-09-23 21:40:57,029 - DEBUG :   time imported: None
2015-09-23 21:40:57,037 - DEBUG :   logging imported: C:\Anaconda3\Lib\logging\__init__.py
2015-09-23 21:40:57,039 - DEBUG :   traceback imported: C:\Anaconda3\Lib\traceback.py
2015-09-23 21:40:57,047 - DEBUG :   ConfigParser imported: C:\Anaconda3\Lib\configparser.py
2015-09-23 21:40:57,050 - DEBUG :   StringIO imported: None
2015-09-23 21:40:57,054 - DEBUG :   datetime imported: C:\Anaconda3\Lib\datetime.py
2015-09-23 21:40:57,057 - DEBUG :   thread imported: None
2015-09-23 21:40:57,059 - DEBUG :   Detected Python3
2015-09-23 21:40:57,065 - DEBUG :   imp imported: C:\Anaconda3\Lib\imp.py
2015-09-23 21:40:57,068 - DEBUG :   Setting coinit_flags=COINIT_APARTMENTTHREADED
2015-09-23 21:40:57,070 - DEBUG :   Creating pyxll logger
2015-09-23 21:40:57,073 - DEBUG :   Getting original stdout/stderr
2015-09-23 21:40:57,075 - DEBUG :   Declaring PyXLLConfigParser
2015-09-23 21:40:57,079 - DEBUG :   Declaring _load_config
2015-09-23 21:40:57,081 - DEBUG :   Declaring _init module function
2015-09-23 21:40:57,084 - DEBUG :   Declaring _open module function

edit: adding a regular python terminal example:
Quote
>>> import site
>>> print(site.__file__)
C:\Anaconda3\lib\site.py

« Last Edit: September 23, 2015, 04:01:53 PM by aavanian »

Tony

  • Administrator
  • Hero Member
  • *****
  • Posts: 581
  • Karma: +13/-1
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #5 on: September 23, 2015, 04:06:13 PM »
Great, that's the script that is supposed to set up your sys.path by reading the .pth files.

It should look in <python>/lib/site-packages where <python> is sys.prefix or sys.exec_prefix (see getsitepackages in site.py) and process any .pth files it finds in those folders (see addsitedir).

So, possibly one of two things is going wrong... either sys.prefix and or sys.exec_prefix is not giving the correct path, or the .pth file isn't where site.py is expecting it to be.

Could you try printing out sys.prefix and sys.exec_prefix to check they point to your Anaconda install please? If they don't you can use a pyxll-startup.py file in the same folder as pyxll.xll to modify them before site.py is imported, eg

Code: [Select]
import sys
sys.prefix = <your sys.prefix>
sys.exec_prefix = <your sys.exec_prefix>

Alternatively you could add some print statements to your site.py to see exactly what .pth files it's finding and whether it's looking in the right place for your one or not.

Best regards,
Tony

aavanian

  • Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #6 on: September 23, 2015, 04:51:58 PM »
So indeed sys.prefix and sys.exec_prefix are empty when the interpreter loads from pyxll.
I added the pyxll-startup.py file as advised and even though the two variables are now correct, the path is still the same, so I guess it's about the .pth files...

I tried adding prints in site.py but those do not appear in any of pyxll logs and not in a cmd prompt when I launch excel from it. I'll try later printing to a file.

Regards,
Alexandre

aavanian

  • Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #7 on: September 24, 2015, 06:40:14 AM »
Ok, so here are the results of my investigations:
The interpreter launched by PyXll loads the correct site.py but actually bypasses all its execution because it is launched with the no_site flag that site.py tests before execution:
Quote
sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=1, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=0, isolated=0)
Quote
if not sys.flags.no_site:
    main()

As I couldn't override the sys.flag.no_site value (it is read-only), I circumvent the issue by importing a modified copy of site.py from pyxll-startup.py in which i disabled the no_site check.
With this, my project now works as intended:
  • In pyxll.cfg, i specify the path to the modules that expose excel functions (I'm not totally sure why I still need to do that but I suspect this is due to PyXll binding more than sys.path issues so fair enough) and the associated modules
  • I also left the paths to the python distribution (as per the anaconda/pyxll installation instructions) but I suspect this is not necessary anymore
  • The modules exposing excel functions can import other modules of my project (independently of their location relative to the modules directly imported by PyXll)
But this is definitely an ugly hack.

From there, a few questions:
  • For which purpose does PyXll launch the interpreter with no_site enabled?
  • Any side effect to be expected from my hack?
  • Would it be possible to launch the interpreter without no_site? or leave that as a configuration flag to be specified somewhere?
  • Absent that, do you have any idea for a better solution than what I'm currently doing?

Thanks in advance,
Alexandre

PS: your site certificate has expired!

Tony

  • Administrator
  • Hero Member
  • *****
  • Posts: 581
  • Karma: +13/-1
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #8 on: September 24, 2015, 10:09:21 AM »
Hi Alexandre,

thanks very much for looking into this and your thorough explanation of what's going wrong.

no_site is an interpreter flag that tells Python to import site.py when the interpreter is initialized. When PyXLL is initialized the import of site.py is deferred to allow pyxll-startup to be imported first. This seems like an oversight in a newer version of site.py (previously, and in the version I have been testing against) this check was not there as it's not needed (since it's just supposed to tell the interpreter whether to import site.py or not, not change the behavior of site.py once it is imported). I will look at how PyXLL can be modified to handle this change to site.py.

Perhaps a cleaner work around for now would be to import site.py in your pyxll-startup.py and just call main?

Code: [Select]
import site
site.main()

Having said that I don't see any problem with your hack (but I will resolve this no_site issue so it's not necessary).

Now the python path is being set correctly you shouldn't need to add the path explicitly to your config file. When PyXLL looks for the modules listed in the config file it looks on the python path, and it shouldn't matter if that comes from the default path, or added by site.py, or is included in the config. If the modules you want it to import are part of a package you can use the full module name in the config (e.g. mypackage.mymodule).

Thanks again for your help diagnosing this.

Best regards,
Tony

ps. yes, sorry about that :( Mix up with the hosting provider that will hopefully be resolved soon!

 

aavanian

  • Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #9 on: September 24, 2015, 12:09:09 PM »
/facepalm, indeed, site.main() looks much better. Not sure I can import site this way though (for my hack, I need to import my modified site.py with the filepath) but will try.

Thanks again for the quick turn-arounds.
Alexandre

Tony

  • Administrator
  • Hero Member
  • *****
  • Posts: 581
  • Karma: +13/-1
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #10 on: September 28, 2015, 09:06:01 AM »
Hi Alexandre,

I've fixed the no_site problem. You can download the patched version from https://beta.pyxll.com/download.html.

Please could you let me know if this resolves your issue?

thanks!
Tony

aavanian

  • Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #11 on: September 29, 2015, 10:14:29 AM »
Hi,

I could not use the xll in the new beta, getting a corrupted file warning in excel

Regards
Alexandre

Tony

  • Administrator
  • Hero Member
  • *****
  • Posts: 581
  • Karma: +13/-1
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #12 on: September 29, 2015, 11:08:29 AM »
Hi Alexandre,

looks like you've downloaded the x86 build, whereas previously you were using the x64 version. Please can you try again with the x64 build?

cheers,
Tony

aavanian

  • Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #13 on: September 30, 2015, 01:49:27 AM »
My bad. It does solve the issue.

Thanks
Alexandre

Tony

  • Administrator
  • Hero Member
  • *****
  • Posts: 581
  • Karma: +13/-1
    • View Profile
Re: eggs in site-packages not imported with latest Anaconda
« Reply #14 on: September 30, 2015, 07:51:45 AM »
Great. Thanks again for your help identifying this problem.

cheers,
Tony