Lc@sdZddkZddkZddkZddkZddkZddkZddkZddkZddk Z ddk Z ddk Z ddk Z ddk Z yddklZWn#ej oddklZnXddklZlZlZlZlZlZlZlZlZlZlZlZddklZlZl Z l!Z!e i"d Z#da%de i&dZ'dZ(d e)fd YZ*d e*efd YZ+ei,d Z-dZ.ddCdYZ/ddDdYZ0dZ1ddEdYZ2de2fdYZ3de2fdYZ4de2fdYZ5dZ6de2fdYZ7ddFd YZ8d!e8fd"YZ9d#dGd$YZ:d%e:e2fd&YZ;d'e:e2fd(YZ<d)Z=d*dHd+YZ>d,e2e>fd-YZ?d.e2e>fd/YZ@d0e2fd1YZAd2eAfd3YZBeCed4od5eAfd6YZDnd7e2fd8YZEd9e2fd:YZFd;ZGd<ZHd=e2fd>YZId?e2fd@YZJdAeJfdBYZKdS(Is! An extensible library for opening URLs using a variety of protocols The simplest way to use this module is to call the urlopen function, which accepts a string containing a URL or a Request object (described below). It opens the URL and returns the results as file-like object; the returned object has some extra methods described below. The OpenerDirector manages a collection of Handler objects that do all the actual work. Each Handler implements a particular protocol or option. The OpenerDirector is a composite object that invokes the Handlers needed to open the requested URL. For example, the HTTPHandler performs HTTP GET and POST requests and deals with non-error returns. The HTTPRedirectHandler automatically deals with HTTP 301, 302, 303 and 307 redirect errors, and the HTTPDigestAuthHandler deals with digest authentication. urlopen(url, data=None) -- Basic usage is the same as original urllib. pass the url and optionally data to post to an HTTP URL, and get a file-like object back. One difference is that you can also pass a Request instance instead of URL. Raises a URLError (subclass of IOError); for HTTP errors, raises an HTTPError, which can also be treated as a valid response. build_opener -- Function that creates a new OpenerDirector instance. Will install the default handlers. Accepts one or more Handlers as arguments, either instances or Handler classes that it will instantiate. If one of the argument is a subclass of the default handler, the argument will be installed instead of the default. install_opener -- Installs a new opener as the default opener. objects of interest: OpenerDirector -- Sets up the User Agent as the Python-urllib client and manages the Handler classes, while dealing with requests and responses. Request -- An object that encapsulates the state of a request. The state can be as simple as the URL. It can also include extra HTTP headers, e.g. a User-Agent. BaseHandler -- exceptions: URLError -- A subclass of IOError, individual protocols have their own specific subclass. HTTPError -- Also a valid HTTP response, so you can treat an HTTP error as an exceptional event or valid response. internals: BaseHandler and parent _call_chain conventions Example usage: import urllib2 # set up authentication info authinfo = urllib2.HTTPBasicAuthHandler() authinfo.add_password(realm='PDQ Application', uri='https://mahler:8092/site-updates.py', user='klem', passwd='geheim$parole') proxy_support = urllib2.ProxyHandler({"http" : "http://ahad-haam:3128"}) # build a new opener that adds authentication and caching FTP handlers opener = urllib2.build_opener(proxy_support, authinfo, urllib2.CacheFTPHandler) # install it urllib2.install_opener(opener) f = urllib2.urlopen('http://www.python.org/') iN(tStringIO( tunwraptunquotet splittypet splithosttquotet addinfourlt splitportt splitattrt ftpwrappert splitusert splitpasswdt splitvalue(t localhostt url2pathnamet getproxiest proxy_bypassicCs-tdjo tanti|||S(N(t_openertNonet build_openertopen(turltdatattimeout((s/usr/lib64/python2.6/urllib2.pyturlopenzs  cCs |adS(N(R(topener((s/usr/lib64/python2.6/urllib2.pytinstall_openerstURLErrorcBseZdZdZRS(cCs|f|_||_dS(N(targstreason(tselfR((s/usr/lib64/python2.6/urllib2.pyt__init__s cCs d|iS(Ns(R(R((s/usr/lib64/python2.6/urllib2.pyt__str__s(t__name__t __module__RR (((s/usr/lib64/python2.6/urllib2.pyRs t HTTPErrorcBs)eZdZeiZdZdZRS(sBRaised when HTTP error occurs, but also acts like non-error returncCsX||_||_||_||_||_|dj o|i||||ndS(N(tcodetmsgthdrstfptfilenameRt_HTTPError__super_init(RRR$R%R&R'((s/usr/lib64/python2.6/urllib2.pyRs      cCsd|i|ifS(NsHTTP Error %s: %s(R$R%(R((s/usr/lib64/python2.6/urllib2.pyR s(R!R"t__doc__RRR)R (((s/usr/lib64/python2.6/urllib2.pyR#s  s:\d+$cCsa|i}ti|d}|djo|idd}ntid|d}|iS(sReturn request-host, as defined by RFC 2965. Variation from RFC: returned value is lowercased, for convenient comparison. ittHost(t get_full_urlturlparset get_headert _cut_port_retsubtlower(trequestRthost((s/usr/lib64/python2.6/urllib2.pyt request_hosts   tRequestcBseZdhdedZdZdZdZdZdZ dZ dZ dZ d Z d Zd Zd Zd ZdZdZdZddZdZRS(cCst||_d|_d|_d|_d|_||_h|_x*|i D]\}}|i ||qRWh|_ |djot |}n||_ ||_dS(N(Rt_Request__originalRttypeR4tportt _tunnel_hostRtheaderstitemst add_headertunredirected_hdrsR5torigin_req_hostt unverifiable(RRRR;R?R@tkeytvalue((s/usr/lib64/python2.6/urllib2.pyRs          cCsb|d djoD|d}ttd|o"t|d|t||Snt|dS(Ni t _Request__r_tget_(thasattrR6tgetattrtAttributeError(Rtattrtname((s/usr/lib64/python2.6/urllib2.pyt __getattr__s  cCs|iodSdSdS(NtPOSTtGET(thas_data(R((s/usr/lib64/python2.6/urllib2.pyt get_methods cCs ||_dS(N(R(RR((s/usr/lib64/python2.6/urllib2.pytadd_datascCs |idj S(N(RR(R((s/usr/lib64/python2.6/urllib2.pyRMscCs|iS(N(R(R((s/usr/lib64/python2.6/urllib2.pytget_datascCs|iS(N(R7(R((s/usr/lib64/python2.6/urllib2.pyR-scCsZ|idjoCt|i\|_|_|idjotd|iqSn|iS(Nsunknown url type: %s(R8RRR7t_Request__r_typet ValueError(R((s/usr/lib64/python2.6/urllib2.pytget_types cCsV|idjo?t|i\|_|_|iot|i|_qOn|iS(N(R4RRRQt_Request__r_hostR(R((s/usr/lib64/python2.6/urllib2.pytget_hosts  cCs|iS(N(RT(R((s/usr/lib64/python2.6/urllib2.pyt get_selectorscCsM|idjo|i o|i|_n||_|i|_||_dS(Nthttps(R8R:R4R7RT(RR4R8((s/usr/lib64/python2.6/urllib2.pyt set_proxys   cCs|i|ijS(N(RTR7(R((s/usr/lib64/python2.6/urllib2.pyt has_proxy scCs|iS(N(R?(R((s/usr/lib64/python2.6/urllib2.pytget_origin_req_host scCs|iS(N(R@(R((s/usr/lib64/python2.6/urllib2.pytis_unverifiablescCs||i|iR\(RRAR]((s/usr/lib64/python2.6/urllib2.pytadd_unredirected_headerscCs||ijp ||ijS(N(R;R>(Rt header_name((s/usr/lib64/python2.6/urllib2.pyt has_headerscCs"|ii||ii||S(N(R;tgetR>(RR_tdefault((s/usr/lib64/python2.6/urllib2.pyR/s cCs)|ii}|i|i|iS(N(R>tcopytupdateR;R<(RR&((s/usr/lib64/python2.6/urllib2.pyt header_items$sN(R!R"RtFalseRRJRNRORMRPR-RSRURVRXRYRZR[R=R^R`R/Re(((s/usr/lib64/python2.6/urllib2.pyR6s(               tOpenerDirectorcBsSeZdZdZdZdZdeidZ ddZ dZ RS(cCsMdt}d|fg|_g|_h|_h|_h|_h|_dS(NsPython-urllib/%ss User-agent(t __version__t addheadersthandlerst handle_opent handle_errortprocess_responsetprocess_request(Rtclient_version((s/usr/lib64/python2.6/urllib2.pyR*s     c Cst|dptdt|nt}xst|D]e}|d joq=n|id}|| }||d}|idoq|id|d}||d}yt|}Wntj onX|i i |h} | |i |$s R;R?R@N(i-i.i/i3(sGETR(i-i.i/( RNtreplaceRR;R<R6RZRR#R-( RRR'R$R%R;tnewurltmt newheaders((s/usr/lib64/python2.6/urllib2.pyRqs   c Csd|jo|idd}n)d|jo|idd}ndSti|}|ipt|}d|d>> _parse_proxy('file:/ftp.example.com/') Traceback (most recent call last): ValueError: proxy URL with no authority: 'file:/ftp.example.com/' The first three items of the returned tuple may be None. Examples of authority parsing: >>> _parse_proxy('proxy.example.com') (None, None, None, 'proxy.example.com') >>> _parse_proxy('proxy.example.com:3128') (None, None, None, 'proxy.example.com:3128') The authority component may optionally include userinfo (assumed to be username:password): >>> _parse_proxy('joe:password@proxy.example.com') (None, 'joe', 'password', 'proxy.example.com') >>> _parse_proxy('joe:password@proxy.example.com:3128') (None, 'joe', 'password', 'proxy.example.com:3128') Same examples, but with URLs instead: >>> _parse_proxy('http://proxy.example.com/') ('http', None, None, 'proxy.example.com') >>> _parse_proxy('http://proxy.example.com:3128/') ('http', None, None, 'proxy.example.com:3128') >>> _parse_proxy('http://joe:password@proxy.example.com/') ('http', 'joe', 'password', 'proxy.example.com') >>> _parse_proxy('http://joe:password@proxy.example.com:3128') ('http', 'joe', 'password', 'proxy.example.com:3128') Everything after the authority is ignored: >>> _parse_proxy('ftp://joe:password@proxy.example.com/rubbish:3128') ('ftp', 'joe', 'password', 'proxy.example.com') Test for no trailing '/' case: >>> _parse_proxy('http://joe:password@proxy.example.com') ('http', 'joe', 'password', 'proxy.example.com') Rs//sproxy URL with no authority: %riiN(RRzRRRRyR R ( tproxytschemetr_schemet authoritytendtuserinfothostporttusertpassword((s/usr/lib64/python2.6/urllib2.pyt _parse_proxyqs2      RcBs#eZdZddZdZRS(idcCsg|djo t}n||_x=|iD]/\}}t|d||||idq0WdS(Ns%s_opencSs||||S(((trRR8R((s/usr/lib64/python2.6/urllib2.pyts(RRtproxiesR<tsetattrRs(RRR8R((s/usr/lib64/python2.6/urllib2.pyRs     c Cs|i}t|\}}}}|djo |}n|i|iot|iodS|oP|oIdt|t|f} ti| i } |i dd| nt|}|i ||||jp |djodS|i i |d|iSdS(Ns%s:%ssProxy-authorizationsBasic RWR(RSRRRUR4RRtbase64t b64encodetstripR=RXRRR( RRRR8t orig_typet proxy_typeRRRt user_passtcreds((s/usr/lib64/python2.6/urllib2.pyRss      N(R!R"RRRRs(((s/usr/lib64/python2.6/urllib2.pyRs tHTTPPasswordMgrcBs8eZdZdZdZedZdZRS(cCs h|_dS(N(tpasswd(R((s/usr/lib64/python2.6/urllib2.pyRsc Cst|to |g}n||ijoh|i|s    cCs d|_dS(Ni(R(R((s/usr/lib64/python2.6/urllib2.pytreset_retry_countEsc Cs|i|d}|idjo"t|idd|dn|id7_|o]tii|}|o@|i\}}}|i djo|i |||SqndS(Niisbasic auth faileditbasic( RaRRR#R-RtrxtsearchtgroupsR2tretry_http_basic_auth( RtauthreqR4RR;tmoRRR((s/usr/lib64/python2.6/urllib2.pythttp_error_auth_reqedHscCs|ii||\}}|dj ozd||f}dti|i}|ii|id|jodS|i |i||i i |d|i SdSdS(Ns%s:%ssBasic %sR( RR RRRRR;Rat auth_headerR^RRR(RR4RRRtpwtrawtauth((s/usr/lib64/python2.6/urllib2.pyR\s N( R!R"tretcompiletIRRRRR R(((s/usr/lib64/python2.6/urllib2.pyR0s      tHTTPBasicAuthHandlercBseZdZdZRS(t AuthorizationcCs2|i}|id|||}|i|S(Nswww-authenticate(R-R R(RRR'R$R%R;RRv((s/usr/lib64/python2.6/urllib2.pythttp_error_401ms    (R!R"R!R*(((s/usr/lib64/python2.6/urllib2.pyR(istProxyBasicAuthHandlercBseZdZdZRS(sProxy-authorizationcCs2|i}|id|||}|i|S(Nsproxy-authenticate(RUR R(RRR'R$R%R;RRv((s/usr/lib64/python2.6/urllib2.pythttp_error_407ys    (R!R"R!R,(((s/usr/lib64/python2.6/urllib2.pyR+uscCstiido*td}|i|}|i|Sg}t|D]}|tti ddqN~}di |SdS(sReturn n random bytes.s /dev/urandomiiR+N( tosRtexistsRRRtrangetchrtrandomt randrangetjoin(tntftsRRtL((s/usr/lib64/python2.6/urllib2.pyt randombytess  9tAbstractDigestAuthHandlercBsSeZddZdZdZdZdZdZdZ dZ RS( cCsQ|djo t}n||_|ii|_d|_d|_d|_dS(Ni(RRRRRt nonce_countt last_nonce(RR((s/usr/lib64/python2.6/urllib2.pyRs     cCs d|_dS(Ni(R(R((s/usr/lib64/python2.6/urllib2.pyRscCs|i|d}|idjo"t|idd|dn|id7_|o8|id}|idjo|i||SndS(Niisdigest auth failediitdigest(RaRRR#R-tsplitR2tretry_http_digest_auth(RR!R4RR;RR((s/usr/lib64/python2.6/urllib2.pyR scCs|idd\}}tt|}|i||}|oad|}|ii|id|jodS|i|i||i i |d|i }|SdS(NRis Digest %sR( R=tparse_keqv_listtparse_http_listtget_authorizationR;RaR!RR^RRR(RRR$ttokent challengetchaltauth_valtresp((s/usr/lib64/python2.6/urllib2.pyR>s cCs<tid|i|titdfi}|d S(Ns %s:%s:%s:%sii(thashlibtsha1R:ttimetctimeR8t hexdigest(Rtnoncetdig((s/usr/lib64/python2.6/urllib2.pyt get_cnoncesc CsxyK|d}|d}|id}|idd}|idd}Wntj odSX|i|\}} |djodS|ii||i\} } | djodS|io|i|i |} nd} d| || f} d|i |i f}|d jo||i jo|i d 7_ nd |_ ||_ d |i }|i|}d ||||||f}| || |}nG|djo)| || d|||f}ntd |d| |||i |f}|o|d|7}n| o|d| 7}n|d|7}|o|d||f7}n|S(NRRLtqopt algorithmtMD5topaques%s:%s:%ss%s:%sR$is%08xs%s:%s:%s:%s:%ssqop '%s' is not supported.s>username="%s", realm="%s", nonce="%s", uri="%s", response="%s"s , opaque="%s"s , digest="%s"s, algorithm="%s"s, qop=auth, nc=%s, cnonce="%s"(RaRtKeyErrortget_algorithm_implsRR R-RMtget_entity_digestRPRNRVR;R:RNR(RRRDRRLRORPRRtHtKDRR"tentdigtA1tA2tncvaluetcnoncetnoncebittrespdigR((s/usr/lib64/python2.6/urllib2.pyRAsV   !        ) csY|i}|djo dn|djo dnfd}|fS(NRQcSsti|iS((RGtmd5RK(tx((s/usr/lib64/python2.6/urllib2.pyRstSHAcSsti|iS((RGRHRK(R`((s/usr/lib64/python2.6/urllib2.pyRscsd||fS(s%s:%s((R6td(RV(s/usr/lib64/python2.6/urllib2.pyRs(tupper(RRPRW((RVs/usr/lib64/python2.6/urllib2.pyRTs     cCsdS(N(R(RRRD((s/usr/lib64/python2.6/urllib2.pyRUsN( R!R"RRRR R>RNRARTRU(((s/usr/lib64/python2.6/urllib2.pyR9s   = tHTTPDigestAuthHandlercBs#eZdZdZdZdZRS(sAn authentication protocol defined by RFC 2069 Digest authentication improves on basic authentication because it does not transmit passwords in the clear. R)icCs?ti|id}|id|||}|i|S(Niswww-authenticate(R.R-R R(RRR'R$R%R;R4tretry((s/usr/lib64/python2.6/urllib2.pyR*)s   (R!R"R*R!RR*(((s/usr/lib64/python2.6/urllib2.pyRdstProxyDigestAuthHandlercBseZdZdZdZRS(sProxy-AuthorizationicCs2|i}|id|||}|i|S(Nsproxy-authenticate(RUR R(RRR'R$R%R;R4Re((s/usr/lib64/python2.6/urllib2.pyR,6s    (R!R"R!RR,(((s/usr/lib64/python2.6/urllib2.pyRf1stAbstractHTTPHandlercBs/eZddZdZdZdZRS(icCs ||_dS(N(t _debuglevel(Rt debuglevel((s/usr/lib64/python2.6/urllib2.pyR?scCs ||_dS(N(Rh(Rtlevel((s/usr/lib64/python2.6/urllib2.pytset_http_debuglevelBsc CsH|i}|ptdn|iob|i}|idp|iddn|idp|iddt|qn|}|io.t|i \}}t |\}}n|idp|id|nxJ|i i D]<\}} |i }|i|p|i|| qqW|S(Ns no host givens Content-types!application/x-www-form-urlencodedsContent-lengths%dR,(RURRMRPR`R^RRYRRVRRRiR\( RR3R4Rtsel_hostRtseltsel_pathRIRB((s/usr/lib64/python2.6/urllib2.pyt do_request_Es0       c Cs|i}|ptdn||d|i}|i|it|i}|i|id|d}s sProxy-AuthorizationR;(RURRtset_debuglevelRhRR;RdR>R<R:t _set_tunnelR3RNRVRt getresponseRRuRtrecvt _fileobjectRRR%R-tstatusR$R( Rt http_classRR4RR;ttunnel_headerstproxy_auth_hdrRterrR'RF((s/usr/lib64/python2.6/urllib2.pyRrbs8     %   (R!R"RRkRoRr(((s/usr/lib64/python2.6/urllib2.pyRg=s   RcBseZdZeiZRS(cCs|iti|S(N(RrRtHTTPConnection(RR((s/usr/lib64/python2.6/urllib2.pyt http_opens(R!R"R}RgRot http_request(((s/usr/lib64/python2.6/urllib2.pyRs RRcBseZdZeiZRS(cCs|iti|S(N(RrRtHTTPSConnection(RR((s/usr/lib64/python2.6/urllib2.pyt https_opens(R!R"RRgRot https_request(((s/usr/lib64/python2.6/urllib2.pyRs tHTTPCookieProcessorcBs2eZddZdZdZeZeZRS(cCs6ddk}|djo|i}n||_dS(Ni(t cookielibRt CookieJart cookiejar(RRR((s/usr/lib64/python2.6/urllib2.pyRs  cCs|ii||S(N(Rtadd_cookie_header(RR3((s/usr/lib64/python2.6/urllib2.pyR~scCs|ii|||S(N(Rtextract_cookies(RR3Rv((s/usr/lib64/python2.6/urllib2.pyRsN(R!R"RRR~RRR(((s/usr/lib64/python2.6/urllib2.pyRs    RcBseZdZRS(cCs |i}td|dS(Nsunknown url type: %s(RSR(RRR8((s/usr/lib64/python2.6/urllib2.pyRs (R!R"R(((s/usr/lib64/python2.6/urllib2.pyRscCsph}xc|D][}|idd\}}|ddjo"|ddjo|dd!}n|||Parse list of key=value strings where keys are not duplicated.t=iit"i(R=(tltparsedteltRR((s/usr/lib64/python2.6/urllib2.pyR?s"cCsg}d}t}}x|D]}|o||7}t}qn|oB|djo t}qn|djo t}n||7}qn|djo|i|d}qn|djo t}n||7}qW|o|i|ng}|D]}||iq~S(spParse lists as described by RFC 2068 Section 2. In particular, parse comma-separated lists where the elements of the list may include quoted-strings. A quoted-string could contain a comma. A non-quoted string could have quotes in the middle. Neither commas nor quotes count if they are escaped. Only double-quotes count, not single-quotes. R+s\Rt,(RfRRR(R6trestparttescapeRtcurR((s/usr/lib64/python2.6/urllib2.pyR@s6           RcBs)eZdZdZdZdZRS(cCsv|i}|d djoH|dd!djo4|io*|idjod|_|ii|S|i|SdS(Nis//iRR tftp(RVR4R8RRtopen_local_file(RRR((s/usr/lib64/python2.6/urllib2.pyt file_opens  / cCstidjoiy7ttiddtitidt_Wqytij otidft_qyXntiS(NR i( RtnamesRRRtgethostbyname_ext gethostnametgaierrort gethostbyname(R((s/usr/lib64/python2.6/urllib2.pyt get_names s$c Csiddk}ddk}|i}|i}t|}yti|}|i}|ii |i dt } |i |d} t itd| pd|| f} |ot|\}} n| p$| oYti||ijo=|od||} n d|} tt|d| | SWn!tj o}t|nXtddS( Nitusegmtis6Content-type: %s Content-length: %d Last-modified: %s s text/plainsfile://trbsfile not on local host(t email.utilst mimetypesRURVRR-tstattst_sizetutilst formatdatetst_mtimeRt guess_typet mimetoolstMessageRRRRRRRtOSErrorR(RRtemailRR4R(t localfiletstatstsizetmodifiedtmtypeR;R9torigurlR%((s/usr/lib64/python2.6/urllib2.pyRs0       $ N(R!R"RRRRR(((s/usr/lib64/python2.6/urllib2.pyRs  RcBseZdZdZRS(cCsddk}ddk}|i}|ptdnt|\}}|djo |i}n t|}t|\}}|ot |\}}nd}t |}t |pd}t |pd}yt i |}Wn$t i j o}t|nXt|i\} } | id} tt | } | d | d} } | o| d o| d} ny9|i||||| |i} | odpd}xP| D]H}t|\}}|id jo|djo|i}qqW| i| |\}}d}|i|id}|o|d|7}n|dj o|djo|d|7}nt|}ti|}t|||iSWn2|ij o#}td|ti dnXdS(Nisftp error: no host givenR+RiiR'tDR8tatARRbsContent-type: %s sContent-length: %d s ftp error: %si(RRRR'RbR(!tftplibRRURRRtFTP_PORTR{R R RRRRuRRVR=tmapt connect_ftpRR R2RctretrfileRR-RRRRt all_errorstsystexc_info(RRRRR4R9RRR%RtattrstdirstfiletfwR8RHRBR'tretrlenR;Rtsf((s/usr/lib64/python2.6/urllib2.pytftp_open3s^       !  cCst||||||}|S(N(R (RRRR4R9RRR((s/usr/lib64/python2.6/urllib2.pyRhs(R!R"RR(((s/usr/lib64/python2.6/urllib2.pyR2s 5tCacheFTPHandlercBs5eZdZdZdZdZdZRS(cCs1h|_h|_d|_d|_d|_dS(Nii<i(tcacheRtsoonesttdelayt max_conns(R((s/usr/lib64/python2.6/urllib2.pyRps     cCs ||_dS(N(R(Rtt((s/usr/lib64/python2.6/urllib2.pyt setTimeoutwscCs ||_dS(N(R(RR((s/usr/lib64/python2.6/urllib2.pyt setMaxConnszscCs|||di||f}||ijoti|i|i|Lsp             R"    n 'i H-@ 9   f +4;