snegov/nevernote
1
0 Fork 0
Code Issues 1 Pull Requests Releases Wiki Activity

Fails if resource is unavailable #2

New Issue
Closed
opened 2019-11-09 10:07:07 +00:00 by snegov · 0 comments
snegov commented 2019-11-09 10:07:07 +00:00
Owner
Copy link
Copy source

Program fails if some resource on a page (or page itself) is unavailable.
In the example below ad hosts are blocked on a router.

(3.8.0/envs/nevernote) ➜  nevernote git:(master) ✗ python main.py http://yandex.ru
Processing URL: http://yandex.ru
New picture: http://yastatic.net/lego/_/La6qi18Z8LwgnZdsAr1qy1GwCwo.gif
New picture: https://avatars.mds.yandex.net/get-banana/64225/x25B_1P58Ic8_cTtD_HSabq9u_banana_20161021_gramota1.png/optimize
New picture: https://mc.yandex.ru/watch/722545
Traceback (most recent call last):
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn
    conn = connection.create_connection(
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/Users/snegovik/.pyenv/versions/3.8.0/lib/python3.8/socket.py", line 914, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 334, in connect
    conn = self._new_conn()
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 168, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mc.yandex.ru', port=443): Max retries exceeded with url: /watch/722545 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 144, in <module>
    sys.exit(main())
  File "main.py", line 140, in main
    process_url(arg, dup_check=args.dup_check)
  File "main.py", line 90, in process_url
    img_b64 = get_embedded_binary(img_url)
  File "main.py", line 24, in get_embedded_binary
    response = requests.get(url)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mc.yandex.ru', port=443): Max retries exceeded with url: /watch/722545 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
Program fails if some resource on a page (or page itself) is unavailable. In the example below ad hosts are blocked on a router. ``` (3.8.0/envs/nevernote) ➜ nevernote git:(master) ✗ python main.py http://yandex.ru Processing URL: http://yandex.ru New picture: http://yastatic.net/lego/_/La6qi18Z8LwgnZdsAr1qy1GwCwo.gif New picture: https://avatars.mds.yandex.net/get-banana/64225/x25B_1P58Ic8_cTtD_HSabq9u_banana_20161021_gramota1.png/optimize New picture: https://mc.yandex.ru/watch/722545 Traceback (most recent call last): File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn conn = connection.create_connection( File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/Users/snegovik/.pyenv/versions/3.8.0/lib/python3.8/socket.py", line 914, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno 8] nodename nor servname provided, or not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request self._validate_conn(conn) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn conn.connect() File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 334, in connect conn = self._new_conn() File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 168, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen retries = retries.increment( File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mc.yandex.ru', port=443): Max retries exceeded with url: /watch/722545 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "main.py", line 144, in <module> sys.exit(main()) File "main.py", line 140, in main process_url(arg, dup_check=args.dup_check) File "main.py", line 90, in process_url img_b64 = get_embedded_binary(img_url) File "main.py", line 24, in get_embedded_binary response = requests.get(url) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mc.yandex.ru', port=443): Max retries exceeded with url: /watch/722545 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')) ```
snegov self-assigned this 2019-11-09 10:07:28 +00:00
snegov added the bug label 2019-11-09 10:07:59 +00:00
snegov removed their assignment 2019-11-09 10:48:00 +00:00
snegov self-assigned this 2019-11-09 10:48:18 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No labels bug
Milestone
No items
No Milestone
Assignees
Clear assignees
snegov (Maks Snegov)
No Assignees snegov
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: snegov/nevernote#2