Fails if resource is unavailable #2

New Issue

snegov · 2019-11-09T10:07:07Z

snegov commented

2019-11-09 10:07:07 +00:00

Program fails if some resource on a page (or page itself) is unavailable.
In the example below ad hosts are blocked on a router.

(3.8.0/envs/nevernote) ➜  nevernote git:(master) ✗ python main.py http://yandex.ru
Processing URL: http://yandex.ru
New picture: http://yastatic.net/lego/_/La6qi18Z8LwgnZdsAr1qy1GwCwo.gif
New picture: https://avatars.mds.yandex.net/get-banana/64225/x25B_1P58Ic8_cTtD_HSabq9u_banana_20161021_gramota1.png/optimize
New picture: https://mc.yandex.ru/watch/722545
Traceback (most recent call last):
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn
    conn = connection.create_connection(
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/Users/snegovik/.pyenv/versions/3.8.0/lib/python3.8/socket.py", line 914, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 334, in connect
    conn = self._new_conn()
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 168, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mc.yandex.ru', port=443): Max retries exceeded with url: /watch/722545 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 144, in <module>
    sys.exit(main())
  File "main.py", line 140, in main
    process_url(arg, dup_check=args.dup_check)
  File "main.py", line 90, in process_url
    img_b64 = get_embedded_binary(img_url)
  File "main.py", line 24, in get_embedded_binary
    response = requests.get(url)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mc.yandex.ru', port=443): Max retries exceeded with url: /watch/722545 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

Program fails if some resource on a page (or page itself) is unavailable. In the example below ad hosts are blocked on a router. ``` (3.8.0/envs/nevernote) ➜ nevernote git:(master) ✗ python main.py http://yandex.ru Processing URL: http://yandex.ru New picture: http://yastatic.net/lego/_/La6qi18Z8LwgnZdsAr1qy1GwCwo.gif New picture: https://avatars.mds.yandex.net/get-banana/64225/x25B_1P58Ic8_cTtD_HSabq9u_banana_20161021_gramota1.png/optimize New picture: https://mc.yandex.ru/watch/722545 Traceback (most recent call last): File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn conn = connection.create_connection( File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/Users/snegovik/.pyenv/versions/3.8.0/lib/python3.8/socket.py", line 914, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno 8] nodename nor servname provided, or not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request self._validate_conn(conn) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn conn.connect() File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 334, in connect conn = self._new_conn() File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connection.py", line 168, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen retries = retries.increment( File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mc.yandex.ru', port=443): Max retries exceeded with url: /watch/722545 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "main.py", line 144, in <module> sys.exit(main()) File "main.py", line 140, in main process_url(arg, dup_check=args.dup_check) File "main.py", line 90, in process_url img_b64 = get_embedded_binary(img_url) File "main.py", line 24, in get_embedded_binary response = requests.get(url) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/Users/snegovik/.pyenv/versions/3.8.0/envs/nevernote/lib/python3.8/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mc.yandex.ru', port=443): Max retries exceeded with url: /watch/722545 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x103c9a6a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')) ```

snegov self-assigned this 2019-11-09 10:07:28 +00:00

snegov added the

bug

label 2019-11-09 10:07:59 +00:00

snegov removed their assignment 2019-11-09 10:48:00 +00:00

snegov self-assigned this 2019-11-09 10:48:18 +00:00

snegov closed this issue

2019-11-09 15:34:20 +00:00

Sign in to join this conversation.